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FOREWORD 


This  book  is  one  of  many  technical  management  educational  guides  written  from  a  Department  of 
Defense  (DoD)  perspective;  i.e.,  non-Service  peculiar.  They  are  intended  primarily  for  use  in  the  courses 
at  the  Defense  Acquisition  University  and  secondarily  as  a  desk  reference  for  program  and  project 
management  personnel.  These  guidebooks  are  written  for  current  and  potential  acquisition  manage¬ 
ment  personnel  who  are  familiar  with  basic  terms  and  definitions  employed  in  program  offices.  They 
are  designed  to  assist  government  and  industry  personnel  in  executing  their  management  responsibili¬ 
ties  relative  to  the  acquisition  and  support  of  defense  systems. 

The  objective  of  a  well-managed  test  and  evaluation  program  is  to  provide  timely  and  accurate  infor¬ 
mation.  This  guide  has  been  developed  to  assist  the  acquisition  community  in  obtaining  a  better  under¬ 
standing  of  whom  the  decision  makers  are  and  determining  how  and  when  to  plan  test  and  evaluation 
events. 


Dr.  John  D.  Claxton 
Program  Director,  T&E 
Defense  Acquisition  University 
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MODULE 


MANAGEMENT  OF 
TEST  AND  EVALUATION 


Test  and  Evaluation  is  a  management  tool  and  an  integral  part  of 
the  development  process.  This  module  will  address  the  policy 
structure  and  oversight  mechanisms  in  place  for  test  and  evaluation. 


IMPORTANCE  OF 
TEST  AND  EVALUATION 


1.1  INTRODUCTION 

The  Test  and  Evaluation  (T&E)  process  is  an  in¬ 
tegral  part  of  the  Systems  Engineering  Process 
(SEP)  which  identifies  levels  of  performance  and 
assists  the  developer  in  correcting  deficiencies.  It 
is  also  a  significant  element  in  the  decision-mak¬ 
ing  process,  providing  data  supportive  of  trade¬ 
off  analysis,  risk  reduction,  and  requirements  re¬ 
finement.  Programmatic  decisions  on  system 
performance  maturity  and  readiness  to  advance  to 
the  next  phase  of  development  take  into  consider¬ 
ation  demonstrated  performance.  The  issue  of  para¬ 
mount  importance  to  the  Service-member  user  is 
system  performance;  i.e.,  will  it  fulfill  the  mission. 
The  T&E  process  provides  data  to  tell  the  user 
how  well  the  system  is  performing  during  devel¬ 
opment  and  if  it  is  ready  for  fielding.  The  Pro¬ 
gram  Manager  (PM)  must  balance  the  risks  of 
cost,  schedule,  and  performance  to  keep  the  pro¬ 
gram  on  track  to  production  and  fielding.  The  re¬ 
sponsibility  of  decision-making  authorities  centers 
on  assessing  risk  tradeoffs.  “Test  and  evaluation 
shall  be  integrated  throughout  the  defense  acqui¬ 
sition  process.  Test  and  evaluation  shall  be  struc¬ 
tured  to  provide  essential  information  to  decision¬ 
makers,  assess  attainment  of  technical  performance 
parameters,  and  determine  whether  systems  are 
operationally  effective,  suitable,  survivable,  and 
safe  for  intended  use.  The  conduct  of  test  and 
evaluation,  integrated  with  modeling  and  simula¬ 
tion,  shall  facilitate  learning,  assess  technology 
maturity  and  interoperability,  facilitate  integration 
into  fielded  forces,  and  confirm  performance 
against  documented  capability  needs  and  adver¬ 
sary  capabilities  as  described  in  the  system  threat 
assessment.”  (Reference  5) 


1.2  TESTING  AS  A  RISK  MANAGEMENT 
TOOL 

Correcting  defects  in  weapons  has  been  estimated 
to  add  from  10-30  percent  to  the  cost  of  each  item 
(Reference  106).  Such  costly  redesign  and  modi¬ 
fication  efforts  can  be  reduced  if  carefully  planned 
and  executed  T&E  programs  are  used  to  detect 
and  fix  system  deficiencies  sufficiently  early  in 
the  acquisition  process  (Figure  1-1).  Fixes  insti¬ 
tuted  during  early  work  efforts  (System  Integra¬ 
tion)  in  the  System  Development  and  Demonstra¬ 
tion  (SDD)  Phase  cost  significantly  less  than  those 
required  in  later  System  Demonstration  after  the 
Critical  Design  Review  (CDR)  when  most  design 
decisions  have  been  made. 

T&E  results  figure  prominently  in  the  decisions 
reached  at  design  and  milestone  reviews.  How¬ 
ever,  the  fact  that  T&E  results  are  required  at  major 
decision  points  does  not  presuppose  that  T&E  re¬ 
sults  must  always  be  favorable.  The  final  deci¬ 
sion  responsibility  lies  with  the  decision  maker 
who  must  examine  the  critical  issues  and  weigh 
the  facts.  Only  the  decision  maker  can  determine 
the  weight  and  importance  that  is  to  be  attributed 
to  a  system’s  diverse  capabilities  and  shortcom¬ 
ings  and  the  degree  of  risk  that  can  be  willingly 
accepted.  The  decision-making  authority  will  be 
unable  to  make  this  judgment  without  a  solid  base 
of  information  provided  by  T&E.  Figure  1-2  il¬ 
lustrates  the  Life  Cycle  Cost  (LCC)  of  the  system 
and  how  decisions  impact  program  expenditures. 

A  Defense  Science  Board  (DSB)  1999  Task 
Force  focused  on  a  broad  overview  of  the  state 
of  T&E  within  the  Department  of  Defense  (DoD) 
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(Reference  41).  This  group  made  the  following 

observations  to  improve  the  T&E  process: 

•  The  focus  of  T&E  should  be  on  how  to  best 
support  the  acquisition  process; 

•  T&E  planning  with  Operational  Test  (OT)  per¬ 
sonnel  should  start  early  in  the  acquisition  cycle; 

•  Distrust  remains  between  the  development  and 
test  communities; 

•  Contractor  testing,  developmental  testing,  and 
operational  testing  have  some  overlapping 
functions; 

•  Independence  of  evaluation  of  test  data  is  the 
essential  element,  not  the  taking  of  the  data  itself; 

•  Response  to  perceived  test  “failures”  is  often 
inappropriate  and  counterproductive. 


The  DSB  Task  Force  (1983)  developed  a  set  of 
templates  for  use  in  establishing  and  maintaining 
low-risk  programs.  Each  template  described  an 
area  of  risk  and  then  specified  technical  methods 
for  reducing  that  risk.  PMs  and  test  managers  may 
wish  to  consult  these  templates  for  guidance  in  re¬ 
ducing  the  risks  frequently  associated  with  test  pro¬ 
grams.  Sample  risk  management  templates  were 
published  as  DoD  4245.7-M,  Transition  from  De¬ 
velopment  to  Production,  September  1985. 

1.3  THE  T&E  CONTRIBUTION  AT 
MAJOR  MILESTONES 

T&E  progress  is  monitored  by  the  Office  of  the 
Secretary  of  Defense  (OSD)  throughout  the  acqui¬ 
sition  process.  Their  oversight  extends  to  Major 
Defense  Acquisition  Programs  (MDAPs)  or  desig¬ 
nated  acquisitions.  T&E  officials  within  OSD  ren¬ 
der  independent  assessments  to  the  Defense  Ac¬ 
quisition  Board  (DAB),  the  Defense  Acquisition 


PHASE:  CR  TD  SDD  P&D  OPNS  &  SUPPORT 

SOURCE:  Defense  Acquisition  University. _ 

Figure  1-2.  Life-Cycle-Cost  Decision  Impact  and  Expenditures 
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Executive  (DAE),  and  the  Secretary  of  Defense 
(SECDEF)  at  each  system  milestone  review. 
These  assessments  are  based  on  the  following 
T&E  information: 

•  The  Test  and  Evaluation  Master  Plan  (TEMP) 
and  more  detailed  supporting  documents  de¬ 
veloped  by  responsible  Service  activities; 

•  Service  test  agency  reports  and  briefings; 

•  T&E,  Modeling  and  Simulation  (M&S),  and 
data  from  other  sources  such  as  Service  PMs, 
laboratories,  industry  developers,  studies,  and 
analyses. 

At  Milestone  B,  the  OSD  T&E  assessments  re¬ 
flect  an  evaluation  of  system  concepts  and  tech¬ 
nology  alternatives  using  early  performance  pa¬ 
rameter  objectives  and  thresholds  found  in  an 
approved  preliminary  TEMP.  At  Milestone  C,  as¬ 
sessments  include  an  evaluation  of  previously  ex¬ 
ecuted  test  plans  and  test  results.  At  the  Full  Rate 
Production  Decision  Review  (FRPDR),  assess¬ 
ments  include  consideration  of  the  operational  ef¬ 
fectiveness  and  suitability  evaluations  of  weapon 
systems. 

A  primary  contribution  made  by  T&E  is  the  detec¬ 
tion  and  reporting  of  deficiencies  that  may  adversely 
impact  the  performance  capability  or  availability/ 
supportability  of  a  system.  A  deficiency  reporting 
process  is  used  throughout  the  acquisition  process 
to  report,  evaluate,  and  track  system  deficiencies 
and  to  provide  the  impetus  for  corrective  actions 
that  improve  performance  to  desired  levels. 

1.3.1  T&E  Contributions  Prior  to 
Milestone  B 

During  Concept  Refinement  (CR)  and  Technol¬ 
ogy  Development  (TD)  activities  prior  to  Mile¬ 
stone  B,  laboratory  testing  and  M&Ss  are  con¬ 
ducted  by  the  contractors  and  the  development 
agency  to  demonstrate  and  assess  the  capabilities 
of  key  subsystems  and  components.  The  test  and 


simulation  designs  are  based  on  the  operational 
needs  documented  in  the  Initial  Capabilities  Docu¬ 
ment  (ICD).  Studies,  analyses,  simulation,  and  test 
data  are  used  by  the  development  agency  to  ex¬ 
plore  and  evaluate  alternative  concepts  proposed 
to  satisfy  the  user’s  needs.  Also  during  this  period, 
the  Operational  Test  Agency  (OTA)  monitors  CR 
and  TD  activities  to  gather  information  for  future 
T&E  planning  and  to  provide  effectiveness  and 
suitability  input  desired  by  the  PM.  The  OTA  also 
conducts  Early  Operational  Assessments  (EOAs), 
as  feasible,  to  assess  the  operational  impact  of  can¬ 
didate  technical  approaches  and  to  assist  in  select¬ 
ing  preferred  alternative  system  concepts. 

The  development  agency  prepares  the  Technol¬ 
ogy  Development  Strategy  (TDS)  with  its  early 
T&E  strategy  for  Development  Test  and  Evalua¬ 
tion  (DT&E)  and  M&S.  The  TDS  presents  T&E 
plans  for  system  design(s)  engineering  and  per¬ 
formance  evaluations.  The  OTA  may  provide  an 
EOA.  This  information  is  incorporated  into  the 
PM’s  TEMP  that  documents  the  program’s  T&E 
strategy  that  supports  the  Milestone  B  decision  to 
proceed  to  the  next  phase. 

1.3.2  T&E  Contributions  Prior  to 
Milestone  C 

During  the  SDD  Phase,  concepts  approved  for 
prototyping  form  the  baseline  that  is  used  for  de¬ 
tailed  test  planning.  The  design  is  matured  into 
an  Engineering  Development  Model  (EDM) 
which  is  tested  in  its  intended  environment  prior 
to  Milestone  C. 

In  Systems  Integration  the  development  agency 
conducts  DT&E  to  assist  with  engineering  design, 
system  development,  risk  identification  and  to  evalu¬ 
ate  the  contractor’s  ability  to  attain  desired  techni¬ 
cal  performance  in  system  specifications  and  achieve 
program  objectives.  The  DT&E  includes  T&E  of 
components,  subsystems,  and  prototype  develop¬ 
ment  models.  T&E  of  functional  compatibility,  in¬ 
teroperability,  and  integration  with  fielded  and  de¬ 
veloping  equipment  and  systems  is  also  included. 
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During  this  phase  of  testing,  adequate  DT&E  is 
accomplished  to  ensure  engineering  is  reasonably 
complete  (including  survivability/vulnerability, 
compatibility,  transportability,  interoperability,  reli¬ 
ability,  maintainability,  safety,  human  factors,  and 
logistic  supportability).  Also,  this  phase  confirms 
that  all  significant  design  problems  have  been  iden¬ 
tified  and  solutions  to  these  problems  are  in  hand. 

The  Service  Operational  Test  and  Evaluation 
(OT&E)  agency  ought  to  conduct  an  EOA  for  the 
Design  Readiness  Review  (DRR)  to  estimate  the 
system’s  potential  to  be  operationally  effective  and 
suitable;  identify  needed  modifications;  and  pro¬ 
vides  information  on  tactics,  doctrine,  organiza¬ 
tion,  and  personnel  requirements.  The  early  OT&E 
program  is  accomplished  in  an  environment  con¬ 
taining  limited  operational  realism.  Typical  opera¬ 
tional  and  support  personnel  are  used  to  obtain 
early  estimates  of  the  user’s  capability  to  operate 
and  maintain  the  system.  Some  of  the  most  im¬ 
portant  products  of  user  assessments  of  system 
maintainability  and  supportability  are  human 
factors  and  safety  issues. 

In  Systems  Demonstration,  the  objective  is  to  de¬ 
sign,  fabricate,  and  test  a  preproduction  system 
that  closely  approximates  the  final  product.  T&E 
activities  of  the  EDM  during  this  period  yield  much 
useful  information.  For  example,  data  obtained 
during  EDM  T&E  can  be  used  to  assist  in  evalu¬ 
ating  the  system’s  maintenance  training  require¬ 
ments  and  the  proposed  training  program.  Test 
results  generated  during  EDM  T&E  also  support 
the  user  in  refining  and  updating  employment 
doctrine  and  tactics. 

During  System  Demonstration,  T&E  is  conducted 
to  satisfy  the  following  objectives: 

(1)  As  specified  in  program  documents,  assess 

the  critical  technical  issues: 

(a)  Determine  how  well  the  development 
contract  specifications  have  been  met; 


(b)  Identify  system  technical  deficiencies  and 
focus  on  areas  for  corrective  actions; 

(c)  Determine  whether  the  system  is  com¬ 
patible,  interoperable,  and  can  be  inte¬ 
grated  with  existing  and  planned  equip¬ 
ment  or  systems; 

(d)  Estimate  the  Reliability,  Availability,  and 
Maintainability  (RAM)  of  the  system 
after  it  is  deployed; 

(e)  Determine  whether  the  system  is  safe; 
ready  for  Low  Rate  Initial  Production 
(LRIP); 

(f)  Evaluate  effects  on  performance  of  any 
configuration  changes  caused  by  cor¬ 
recting  deficiencies,  modifications,  or 
Product  Improvements  (Pis); 

(g)  Assess  human  factors  and  identify 
limiting  factors; 

(2)  Assess  the  technical  risk  and  evaluate  the 
tradeoffs  among  specifications,  operational 
requirements,  LCCs,  and  schedules; 

(3)  Assess  the  survivability,  vulnerability,  and 
logistic  supportability  of  the  system; 

(4)  Verify  the  accuracy  and  completeness  of  the 
technical  documentation  developed  to  main¬ 
tain  and  operate  the  weapons  system; 

(5)  Gather  information  for  training  programs 
and  technical  training  materials  needed  to 
support  the  weapon  system; 

(6)  Provide  information  on  environmental  issues 
for  use  in  preparing  environmental  impact 
assessments; 

(7)  Determine  system  performance  limitations 
and  safe  operating  parameters; 
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Thus,  T&E  activities  intensify  during  this  phase 
and  make  significant  contributions  to  the  overall 
acquisition  decision  process. 

The  development  agency  evaluates  the  results  of 
T&E  for  review  by  the  Service  headquarters  and 
the  Service  acquisition  review  council  prior  to 
system  acquisition  review  by  the  Milestone  Deci¬ 
sion  Authority  (MDA).  The  evaluation  includes 
the  results  of  testing  and  supporting  information, 
conclusions,  and  recommendations  for  further  en¬ 
gineering  development.  At  the  same  time,  the 
OT&E  agency  prepares  an  Independent  Opera¬ 
tional  Assessment  (IOA),  which  contains  estimates 
of  the  system’s  potential  operational  effectiveness 
and  suitability.  The  Operational  Assessment  (OA) 
provides  a  permanent  record  of  OT&E  events,  an 
audit  trail  of  OT&E  data,  test  results,  conclusions, 
and  recommendations.  This  information  is  used 
to  prepare  for  Milestone  C  and  supports  a  recom¬ 
mendation  of  whether  the  design  and  performance 
of  the  system  in  development  justifies  proceeding 
into  LRIP. 

1.3.3  T&E  Contributions  Prior  to  Full  Rate 
Production  Decision  Review 

The  developing  agency  transitions  the  final  de¬ 
sign  to  LRIP  while  fixing  and  verifying  any  tech¬ 
nical  problems  discovered  during  the  final  testing 
of  the  EDM  in  its  intended  environment.  The 
maturity  of  the  hardware  and  software  configura¬ 
tions  and  logistics  support  system  available  from 
LRIP  are  assessed  when  the  developing  agency 
considers  certifying  the  system’s  readiness  for 
Initial  Operational  Test  and  Evaluation  (IOT&E). 

IOT&E  is  conducted  prior  to  the  production  deci¬ 
sion  at  FRPDR  to: 

(1)  Estimate  the  operational  effectiveness  and 
suitability  of  the  system; 

(2)  Identify  operational  deficiencies; 

(3)  Evaluate  changes  in  production  configuration; 


(4)  Provide  information  for  developing  and  re¬ 
fining  logistics  support  requirements  for  the 
system  and  training,  tactics,  techniques,  and 
doctrine; 

(5)  Provide  information  to  refine  O&S  cost  es¬ 
timates  and  identify  system  characteristics  or 
deficiencies  that  can  significantly  impact 
O&S  costs; 

(6)  Determine  whether  the  technical  publica¬ 
tions  and  support  equipment  are  adequate 
in  the  operational  environment. 

Before  the  certification  of  readiness  for  IOT&E, 
the  developer  should  have  obtained  the  Joint  In¬ 
teroperability  Test  Command’s  (JITC’s)  certifica¬ 
tion  of  interoperability  for  the  system  components. 
In  parallel  with  IOT&E,  Live  Fire  Test  and  Evalu¬ 
ation  (LFT&E)  may  be  used  to  evaluate  vulner¬ 
ability  or  lethality  of  a  weapon  system  as  appro¬ 
priate  and  as  required  by  public  law.  The  PM’s 
briefing  and  the  Beyond  Low  Rate  Initial  Pro¬ 
duction  (BLRIP)  report  address  the  risks  of  pro¬ 
ceeding  into  Full  Rate  Production  (FRP). 

1.3.4  T&E  Contributions  After  the  Full  Rate 
Production  Decision 

After  FRPDR,  when  the  FRP  decision  is  normally 
made,  T&E  activities  continue  to  provide  impor¬ 
tant  insights.  Tests  described  in  the  TEMP  but  not 
conducted  during  earlier  phases  are  completed. 
The  residual  DT&E  may  include  extreme  weather 
testing  and  testing  corrected  deficiencies.  System 
elements  are  integrated  into  the  final  operational 
configuration,  and  development  testing  is  com¬ 
pleted  when  all  system  performance  requirements 
are  met.  During  FRP,  government  representatives 
normally  monitor  or  conduct  the  Production  Ac¬ 
ceptance  Test  and  Evaluation  (PAT&E).  Each  sys¬ 
tem  is  verified  by  PAT&E  for  compliance  with 
the  requirements  and  specifications  of  the  contract. 

Post-production  testing  requirements  may  result 
from  an  acquisition  strategy  calling  for  increment 
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changes  to  accommodate  accumulated  engineer¬ 
ing  changes  or  the  application  of  Preplanned  Prod¬ 
uct  Improvements  (P3Is).  This  will  allow  parallel 
development  of  high-risk  technology  and  modu¬ 
lar  insertion  of  system  upgrades  into  production 
equipment.  Technology  breakthroughs  and  signifi¬ 
cant  threat  changes  may  require  system  modifica¬ 
tions.  The  development  of  the  modifications  will 
require  development  testing;  and,  if  system  per¬ 
formance  is  significantly  changed,  some  level  of 
operational  testing  may  be  appropriate. 

OT&E  activities  continue  after  the  FRP  decision 
in  the  form  of  Follow-on  Operational  Test  and 
Evaluation  (FOT&E).  The  initial  phase  of  FOT&E 
may  be  conducted  by  either  the  OT&E  agency  or 
user  commands,  depending  on  Service  directives. 
It  verifies  the  operational  effectiveness  and  suit¬ 
ability  of  the  production  system,  determines  if 
deficiencies  identified  during  the  IOT&E  have 
been  corrected,  and  evaluated  areas  not  tested 
during  IOT&E  due  to  system  limitations.  Addi¬ 
tional  FOT&E  may  be  conducted  over  the  life  of 
the  system  to  refine  doctrine,  tactics,  techniques, 
and  training  programs  and  evaluate  future  incre¬ 
ments,  modifications,  and  upgrades. 

The  OT&E  agency  prepares  an  OA  report  at  the 
conclusion  of  each  FOT&E.  This  report  records 
test  results,  describes  the  evaluation  accomplished 
to  satisfy  critical  issues  and  objectives  established 
for  FOT&E  and  documents  its  assessment  of  de¬ 
ficiencies  resolved  after  SDD.  Deficiencies  that 
are  not  corrected  are  recorded. 

A  final  report  on  FOT&E  may  also  be  prepared 
by  the  using  command  test  team  emphasizing  the 
operational  utility  of  the  system  when  operated, 
maintained,  and  supported  by  operational  person¬ 
nel  using  the  concepts  specified  for  the  system. 
Specific  attention  is  devoted  to  the  following: 

(1)  The  degree  to  which  the  system  accom¬ 
plishes  its  missions  when  employed  by  op¬ 
erational  personnel  in  a  realistic  scenario  with 
the  appropriate  organization,  doctrine,  threat 


(including  countermeasures  and  nuclear 
threats),  environment  and  using  tactics  and 
techniques  developed  during  earlier 
FOT&E; 

(2)  The  degree  to  which  the  system  can  be  placed 
in  operational  field  use,  with  specific  evalua¬ 
tions  of  availability,  compatibility,  transport¬ 
ability,  interoperability,  reliability,  wartime 
usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability,  logistics 
supportability,  and  training  requirements; 

(3)  The  conditions  under  which  the  system  was 
tested  including  the  natural  weather  and  cli¬ 
matic  conditions,  terrain  effects,  battlefield 
disturbances,  and  enemy  threat  conditions; 

(4)  The  ability  of  the  system  to  perform  its  re¬ 
quired  functions  for  the  duration  of  a  speci¬ 
fied  mission  profile; 

(5)  System  weaknesses  such  as  the  vulnerabil¬ 
ity  of  the  system  to  exploitation  by  counter¬ 
measures  techniques  and  the  practicality  and 
probability  of  an  adversary  exploiting  the 
susceptibility  of  a  system  in  combat. 

A  specific  evaluation  of  the  personnel  and  logis¬ 
tics  changes  needed  for  the  effective  integration 
of  the  system  into  the  user’s  inventory  is  also 
made.  These  assessments  provide  essential  input 
for  the  later  acquisition  phases  of  the  system 
development  cycle. 

1.4  SUMMARY 

“Risk  management  is  the  means  by  which  the 
program  areas  of  vulnerability  and  concern  are 
identified  and  managed.”  (Reference  19).  T&E 
is  the  discipline  that  helps  to  illuminate  those  ar¬ 
eas  of  vulnerability.  The  importance  of  T&E  in 
the  acquisition  process  is  summarized  well  in  a 
July  2000  General  Accounting  Office  (GAO) 
report  NSIAD-00-199,  Best  Practices:  A  More 
Constructive  Test  Approach  is  Key  to  Better 
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Weapon  System  Outcomes.  They  serve  to  under¬ 
score  the  importance  of  the  T&E  process  as  a 
whole: 

•  Problems  found  late  in  development  signal 
weaknesses  in  testing  and  evaluation; 

•  Testing  early  to  validate  product  knowledge  is 
a  best  practice;  and 

•  Different  incentives  make  testing  a  more  con¬ 
structive  factor  in  commercial  programs  than 
in  weapon  system  programs; 


“To  lessen  the  dependence  on  testing  late  in  de¬ 
velopment  and  to  foster  a  more  constructive  rela¬ 
tionship  between  program  managers  and  testers, 
GAO  recommends  that  the  Secretary  of  Defense 
instruct  acquisition  managers  to  structure  test  plans 
around  the  attainment  of  increasing  levels  of  prod¬ 
uct  maturity,  orchestrate  the  right  mix  of  tools  to 
validate  these  maturity  levels,  and  build  and  re¬ 
source  acquisition  strategies  around  this  approach. 
GAO  also  recommends  that  validation  of  lower 
levels  of  product  maturity  not  be  deferred  to  the 
third  level.  Finally,  GAO  recommends  that  the 
Secretary  require  that  weapon  systems  demon¬ 
strate  a  specified  level  of  product  maturity  before 
major  programmatic  approvals.” 
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THE  TEST  AND 
EVALUATION  PROCESS 


2.1  INTRODUCTION 

The  fundamental  purpose  of  Test  and  Evaluation 
(T&E)  in  a  defense  system’s  development  and 
acquisition  program  is  to  identify  the  areas  of  risk 
to  be  reduced  or  eliminated.  During  the  early 
phases  of  development,  T&E  is  conducted  to  dem¬ 
onstrate  the  feasibility  of  conceptual  approaches, 
evaluate  design  risk,  identify  design  alternatives, 
compare  and  analyze  tradeoffs,  and  estimate  sat¬ 
isfaction  of  operational  requirements.  As  a  system 
undergoes  design  and  development,  the  iterative 
process  of  testing  moves  gradually  from  a  con¬ 
centration  on  Development  Test  and  Evaluation 
(DT&E),  which  is  concerned  chiefly  with  attain¬ 
ment  of  engineering  design  goals,  to  ever  more 
comprehensive  Operational  Test  and  Evaluation 
(OT&E),  which  focuses  on  questions  of  opera¬ 
tional  effectiveness,  suitability,  and  survivability. 
Although  there  are  usually  separate  development 
test  [DT]  and  Operational  Test  [OT]  events,  DT&E 
and  OT&E  are  not  necessarily  serial  phases  in  the 
evolution  of  a  weapon  system  development.  Com¬ 
bined  or  concurrent  development  test  and  opera¬ 
tional  test  is  encouraged  when  appropriate  (pos¬ 
sible  cost/time  savings)  (Reference  16). 

T&E  has  its  origins  in  the  testing  of  hardware. 
This  tradition  is  heavily  embedded  in  its  vocabu¬ 
lary  and  procedures.  The  advent  of  software¬ 
intensive  systems  has  brought  new  challenges  to 
testing  and  new  approaches  are  discussed  in  Chap¬ 
ter  17  of  this  guide.  Remaining  constant  through¬ 
out  the  T&E  process,  whether  testing  hardware  or 
software,  is  the  need  for  thorough,  logical,  system¬ 
atic,  and  early  test  planning  including  feedback  of 
well-documented  and  unbiased  T&E  results  to  sys¬ 
tem  developers,  users,  and  decision  makers. 


T&E  has  many  useful  functions  and  provides  in¬ 
formation  to  many  customers.  The  T&E  gives  in¬ 
formation  to:  developers  for  identifying  and  re¬ 
solving  technical  difficulties;  decision  makers 
responsible  for  procuring  a  new  system  and  for 
the  best  use  of  limited  resources;  and  to  opera¬ 
tional  users  for  refining  requirements  and  support¬ 
ing  development  of  effective  tactics,  doctrine,  and 
procedures. 

2.2  DEFENSE  SYSTEM  ACQUISITION 
PROCESS 

The  defense  system  acquisition  process  was  re¬ 
vised  significantly  in  2003  to  change  its  primary 
objective  to  acquisition  of  quality  products  that 
satisfy  user  needs  with  measurable  improvements 
to  mission  capability  and  operational  support,  in  a 
timely  manner,  and  at  a  fair  and  reasonable  price. 
As  it  is  now  structured,  the  defense  system  life 
cycle  is  to  replicate  the  preferred  acquisition  strat¬ 
egy  of  an  Evolutionary  Acquisition  (EA)  process 
that  uses  either  incremental  or  spiral  development 
processes.  The  three  major  elements,  pre-system 
acquisition,  system  acquisition,  and  sustainment 
may  include  the  following  five  phases: 

(1)  Concept  Refinement  (CR) 

(2)  Technology  Development  (TD) 

(3)  System  Development  and  Demonstration 
(SDD) 

(4)  Production  and  Deployment  (P&D) 

(5)  Operations  and  Support  (O&S). 
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Figure  2-1.  Testing  and  the  Acquisition  Process 


As  Figure  2-1  shows,  these  phases  are  separated 
by  key  decision  points  when  a  Milestone  Deci¬ 
sion  Authority  (MDA)  reviews  a  program  and 
authorizes  advancement  to  the  next  phase  in  the 
cycle.  Thus  T&E  planning  and  test  results  play 
an  important  part  in  the  MDA  review  process. 

The  following  brief  description  of  the  defense 
system  acquisition  process  shows  how  T&E  fits 
within  the  context  of  the  larger  process.  The  de¬ 
scription  is  based  primarily  upon  information  found 
in  Department  of  Defense  Instruction  (DoDI) 
5000.2  Operation  of  the  Defense  Acquisition 
System,  12  May  2003. 


2.2.1  Concept  Refinement  (CR)  and 
Technology  Development  (TD) 

A  defense  pre-system  acquisition  process  begins 
with  the  requirements  process  identification  of  a 
material  need  and  the  development  of  the  Initial 
Capabilities  Document  (ICD).  A  decision  is  made 
to  start  CR  and  the  Technology  Development 
Strategy  (TDS)  evolves  with  the  initial  test  plan¬ 
ning  concepts.  CR  ends  when  the  MDA  approves 
the  preferred  solution  resulting  from  the  Analysis 
of  Alternatives  (AoA),  identifying  Measures  of 
Effectiveness  (MOEs),  and  approves  the  associ¬ 
ated  TDS.  The  TD  Phase  follows  a  Milestone  A 
decision  during  which  the  risks  of  the  relevant 
technologies  of  the  alternative  selected  for  satis¬ 
fying  the  user’s  needs  are  investigated.  Shortly  after 
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the  milestone  decision,  an  integrated  team  begins 
transitioning  the  test  planning  in  the  TDS  into  the 
evaluation  strategy  for  formulation  of  a  Test  and 
Evaluation  Master  Plan  (TEMP).  The  TD  effort 
concludes  with  a  decision  review  at  Milestone  B 
when  an  affordable  increment  of  militarily  useful 
capability  has  been  identified,  the  technology  for 
that  increment  has  been  demonstrated  in  a  relevant 
environment,  and  a  system  can  be  developed  for 
production  within  a  short  time  frame  (normally 
less  than  five  years)  or  the  MDA  decides  to  ter¬ 
minate  the  effort.  Some  of  the  T&E  related  docu¬ 
ments  one  might  see  at  the  Milestone  B  review 
are  the  Acquisition  Decision  Memorandum 
(ADM)  (exit  criteria),  ICD,  Capability  Develop¬ 
ment  Document  (CDD)  (performance  parameters), 
Acquisition  Strategy,  System  Threat  Assessment 
(STA),  an  Early  Operational  Assessment  (EOA), 
and  the  TEMP.  Additional  program  management 
documents  prepared  before  Milestone  B  include: 
the  AoA,  Independent  Cost  Estimate  (ICE),  and 
Concept  Baseline  version  of  the  Acquisition  Pro¬ 
gram  Baseline  (APB),  which  summarizes  the 
weapon’s  functional  specifications,  performance 
parameters,  and  cost  and  schedule  objectives. 

The  program  office  for  major  programs  (Office 
of  the  Secretary  of  Defense  (OSD)  oversight)  must 
give  consideration  to  requesting  a  waiver  for  full- 
up  system  level  Live  Fire  Testing  and  identifica¬ 
tion  of  Low  Rate  Initial  Production  (LRIP)  quan¬ 
tities  for  Initial  Operational  Test  and  Evaluation 
(IOT&E). 

2.2.2  System  Development  and 
Demonstration  (SDD) 

The  Milestone  B  decision  is  program  initiation  for 
systems  acquisition  and  establishes  broad  objec¬ 
tives  for  program  cost,  schedule,  and  technical 
performance.  After  the  Milestone  B  decision  for  a 
program  start,  the  System  Integration  work  effort 
begins  during  which  selected  concept,  typically 
brassboard  or  early  prototype,  is  refined  through 
systems  engineering  and  analysis.  This  work  ef¬ 
fort  ends  when  the  integration  of  the  system 


components  has  been  demonstrated  through  ad¬ 
equate  developmental  testing  of  prototypes.  The 
Design  Readiness  Review  (DRR)  decision  evalu¬ 
ates  design  maturity  and  readiness  to  either  enter 
into  System  Demonstration  or  make  a  change  to 
the  acquisition  strategy.  The  System  Demonstra¬ 
tion  work  effort  is  intended  to  demonstrate  the 
ability  of  the  system  to  operate  in  a  useful  way 
consistent  with  the  approved  Key  Performance  Pa¬ 
rameters  (KPPs).  Work  advances  the  design  to  an 
Engineering  Development  Model  (EDM)  that  is 
evaluated  for  readiness  to  enter  LRIP.  “Success¬ 
ful  development  test  and  evaluation  to  assess  tech¬ 
nical  progress  against  critical  technical  parameters, 
early  operational  assessments,  and,  where  proven 
capabilities  exist,  the  use  of  modeling  and  simula¬ 
tion  to  demonstrate  system  integration  are  critical 
during  this  effort.  Deficiencies  encountered  in  test¬ 
ing  prior  to  MS  [Milestone]  C  shall  be  resolved 
prior  to  proceeding  beyond  LRIP.”  (Reference  16) 
The  EDM  should  have  demonstrated  acceptable 
performance  in  DT&E  and  the  OA,  with  accept¬ 
able  interoperability  and  operational  supportability. 

Preparation  for  the  Milestone  C  decision  estab¬ 
lishes  more  refined  cost,  schedule,  and  perfor¬ 
mance  objectives  and  thresholds  and  the  Capabil¬ 
ity  Production  Document  (CPD)  establishes  the 
criteria  for  testing  of  LRIP  items.  Documents  in¬ 
teresting  to  the  T&E  manager  at  the  time  of  the 
Milestone  C  review  include  the  ADM  (exit  crite¬ 
ria),  updated  TEMP,  updated  STA,  AoA,  Devel¬ 
opment  Baseline,  development  testing  results  and 
the  OA. 

2.2.3  Production  and  Deployment 

During  the  LRIP  work  effort,  the  purpose  is  to 
achieve  an  operational  capability  that  satisfies  mis¬ 
sion  needs.  The  selected  system  design  and  its 
principal  items  of  support  are  fabricated  as  pro¬ 
duction  configuration  models.  Test  articles  nor¬ 
mally  are  subjected  to  qualification  testing,  full- 
up  Live  Fire  Testing  and  IOT&E.  This  work  effort 
ends  with  the  Full  Rate  Production  Decision  Re¬ 
view  (FRPDR)  marking  entry  into  Full  Rate 
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Production  (FRP)  and  deployment  of  the  system 
for  Initial  Operational  Capability  (IOC).  Key  docu¬ 
ments  for  the  T&E  manager  at  the  time  of  the 
FRPDR  are  the  updated  TEMP,  development  test¬ 
ing  results,  the  Service  IOT&E  report,  and  Live 
Fire  Test  Report.  For  Acquisition  Category 
(ACAT)  I  and  designated  oversight  programs,  the 
Director  of  Operational  Test  and  Evaluation 
(DOT&E)  is  required  by  law  to  document  his  as¬ 
sessment  of  the  adequacy  of  IOT&E  and  the  re¬ 
ported  operational  effectiveness  and  suitability  of 
the  system.  This  is  done  in  the  Beyond  LRIP 
(BLRIP)  report.  Also  mandated  by  law  is  the  re¬ 
quirement  for  the  DOT&E  to  submit  the  Live  Fire 
Test  Report  prior  to  the  program  proceeding  be¬ 
yond  LRIP.  These  DOT&E  Reports  may  be  sub¬ 
mitted  as  a  single  document. 

2.2.4  Operations  and  Support 

The  production  continues  at  full  rate  allowing 
continued  deployment  of  the  system  to  operating 
locations  and  achievement  of  Full  Operational 
Capability  (FOC).  This  phase  may  include  major 
modifications  to  the  production  configuration,  in¬ 
crement  upgrades,  and  related  Follow-on  Opera¬ 
tional  Test  and  Evaluation  (FOT&E)  (Operational 
Assessment  (OA)).  Approval  for  major  modifica¬ 
tions  should  identify  the  actions  and  resources 
needed  to  achieve  and  maintain  operational  readi¬ 
ness  and  support  objectives.  The  high  cost  of 
changes  may  require  initiation  of  the  modification 
as  a  new  program.  To  determine  whether  major 
upgrades/modifications  are  necessary  or  deficien¬ 
cies  warrant  consideration  of  replacement,  the 
MDA  may  review  the  impact  of  proposed  changes 
on  system  operational  effectiveness,  suitability,  and 
readiness. 

2.3  T&E  AND  THE  SYSTEMS 

ENGINEERING  PROCESS  (SEP) 

In  the  early  1970s,  DoD  test  policy  became  more 
formalized  and  placed  greater  emphasis  on  T&E 
as  a  continuing  function  throughout  the  acquisition 
cycle.  These  policies  stressed  the  use  of  T&E  to 


reduce  acquisition  risk  and  provide  early  and  con¬ 
tinuing  estimates  of  system  operational  effective¬ 
ness  and  operational  suitability.  To  meet  these  ob¬ 
jectives,  appropriate  test  activities  had  to  be  fully 
integrated  into  the  overall  development  process. 
From  a  systems  engineering  perspective,  test  plan¬ 
ning,  testing,  and  analysis  of  test  results  are  inte¬ 
gral  parts  of  the  basic  product  definition  process. 

Systems  engineering  has  been  defined  in  the 
DoD  context:  Systems  engineering  is  an  inter¬ 
disciplinary  approach  to  evolve  and  verify  an 
integrated  and  optimally  balanced  set  of  prod¬ 
uct  and  process  designs  that  satisfy  user  needs 
and  provide  information  for  management  deci¬ 
sion  making  (Figure  2-2). 

A  system’s  life  cycle  begins  with  the  user’s  needs, 
which  are  expressed  as  constraints,  and  the  re¬ 
quired  capabilities  needed  to  satisfy  mission  ob¬ 
jectives.  Systems  engineering  is  essential  in  the 
earliest  planning  period,  in  conceiving  the  system 
concept  and  defining  performance  requirements 
for  system  elements.  As  the  detailed  design  is  pre¬ 
pared,  systems  engineers  ensure  balanced  influ¬ 
ence  of  all  required  design  specialties,  including 
‘testability.’  They  resolve  interface  problems,  per¬ 
form  design  reviews,  perform  trade-off  analyses, 
and  assist  in  verifying  performance. 

The  days  when  one  or  two  individuals  could  de¬ 
sign  a  complex  system,  especially  a  huge,  mod¬ 
em-age  weapon  system,  are  in  the  past.  Now  sys¬ 
tems  are  too  complex  for  a  small  number  of 
generalists  to  accommodate;  they  require  too  much 
in-depth  knowledge  over  a  broad  range  of  areas 
and  technical  disciplines.  System  engineers  coor¬ 
dinate  the  many  specialized  engineers  involved  in 
the  concurrent  engineering  process  through  Inte¬ 
grated  Product  and  Process  Development  (IPPD). 
Integrated  Product  Teams  (IPTs)  are  responsible 
for  the  integration  of  the  components  into  a 
system. 

Through  interdisciplinary  integration,  a  systems 
engineer  manages  the  progress  of  product  definition 
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PROCESS  INPUT - ► 

•  CUSTOMER  NEEDS/OBJECTIVES/ 
REQUIREMENTS 

•MISSIONS 

•  MEASURES  OF  EFFECTIVENESS 
•ENVIRONMENTS 
•CONSTRAINTS 
•TECHNOLOGY  BASE 

•  OUTPUT  FROM  PRIOR  PHASE 

•  PROGRAM  DECISIONS  REQUIREMENTS 

•  REQUIREMENTS  APPUED  THROUGH 
SPECIFICATIONS  AND  STANDARDS 


PROCESS  OUTPUT 

•  PHASE  DEPENDENT 
■DECISION  SUPPORT  DATA 
■SYSTEM  ARCHITECTURE 
•SPECIFICATIONS  AND  BASELINES 


Figure  2-2.  The  Systems  Engineering  Process 

from  system  level  to  configuration-item  level,  de-  2.3.1  The  Systems  Engineering  Process 
tailed  level,  deficiency  correction,  and  modifications/ 

Product  Improvements  (Pis).  Test  results  provide  The  SEP  is  the  iterative  logical  sequence  of  analy- 

feedback  to  analyze  the  design  progress  toward  per-  sis,  design,  test,  and  decision  activities  that  trans- 

formance  goals.  Tools  of  systems  engineering  in-  forms  an  operational  need  into  the  descriptions 

elude  design  reviews,  configuration  management,  required  for  production  and  fielding  of  all  opera- 

simulation,  Technical  Performance  Measurement  tional  and  support  system  elements.  This  process 

(TPM),  trade-off  analysis,  and  specifications.  consists  of  four  activities.  They  include  require¬ 

ments  analysis,  functional  analysis  and  allocation, 
What  happens  during  systems  engineering?  The  synthesis,  and  verification  of  performance  (T&E) 

process  determines  what  specialists  are  required,  which  support  decisions  on  tradeoffs  and  formal- 

what  segments  and  Non-developmental  Items  ize  the  description  of  system  elements.  The  system 

(NDIs)  are  used,  design  performance  limits,  trade-  engineering  plan  is  described  in  Chapter  16  of  the 

off  criteria,  how  to  test,  when  to  test,  how  to  docu-  Defense  Acquisition  University  guide  to  Systems 

ment  (specifications),  and  what  management  con-  Engineering  Fundamentals,  January  2001 . 

trols  to  apply  (TPM  and  design  reviews). 

The  requirements  analysis  activity  is  a  process 
Development  and  operational  testing  support  the  used  by  the  program  office,  in  concert  with  the 

technical  reviews  by  providing  feedback  to  the  user,  to  establish  and  refine  operational  and  de- 

Systems  Engineering  Process  (SEP).  More  infor-  sign  requirements  that  result  in  the  proper  balance 

mation  on  the  reviews  is  contained  in  Chapter  8.  between  performance  and  cost  within  affordability 
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constraints.  Requirements  analysis  shall  be  con¬ 
ducted  iteratively  with  Functional  Analysis/ 
Allocation  (FA/A)  to  develop  and  refine  system 
level  functional  and  performance  requirements,  ex¬ 
ternal  interfaces,  and  provide  traceability  among 
user  requirements  and  design  requirements. 

The  FA/A  activity  identifies  what  the  system,  com¬ 
ponent,  or  part  must  do.  It  normally  works  from 
the  top  downward  ensuring  requirements  trace- 
ability  and  examining  alternative  concepts.  This 
is  done  without  assuming  how  functions  will  be 
accomplished.  The  product  is  a  series  of  alterna¬ 
tive  Functional  Flow  Block  Diagrams  (FFBDs). 
A  functional  analysis  can  be  applied  at  every  level 
of  development.  At  the  system  level,  it  may  be  a 
contractor  or  Service  effort.  During  the  Concept 
Refinement  (CR)  and  Technology  Development 
(TD)  phases,  developmental  testers  assist  the  func¬ 
tional  analysis  activity  to  help  determine  what 
each  component’s  role  will  be  as  part  of  the  sys¬ 
tem  being  developed.  Performance  requirements 
are  allocated  to  system  components. 

The  synthesis  activity  involves  invention  —  con¬ 
ceiving  ways  to  do  each  FFBD  task  —  to  answer 
the  “how”  question.  Next,  the  physical  interfaces 
implied  by  the  “how”  answers,  are  carefully  iden¬ 
tified  (topological  or  temporal).  The  answers  must 
reflect  all  technology  selection  factors.  Synthesis 
tools  include  Requirements  Allocation  Sheets 
(RASs),  which  translate  functional  statements  into 
design  requirements  and  permit  a  long  and  com¬ 
plex  interactive  invention  process  with  control, 
visibility,  and  requirements  traceability.  Develop¬ 
mental  testers  conduct  prototype  testing  to  deter¬ 
mine  how  the  components  will  perform  assigned 
functions  to  assist  this  synthesis  activity. 

The  verification  loop  and  decision  activity  allows 
tradeoff  of  alternative  approaches  to  “how.”  This 
activity  is  conducted  in  accordance  with  decision 
criteria  set  by  higher-level  technical  requirements 
for  such  things  as:  Life  Cycle  Costs  (LCCs);  ef¬ 
fectiveness;  Reliability,  Availability,  and  Maintain¬ 
ability  (RAM);  risk  limits;  schedule,  etc.  It  is 


repeated  at  each  level  of  development.  The  verifi¬ 
cation  and  decision  activity  is  assisted  by  devel¬ 
opmental  testers  during  the  later  SDD  phase  when 
competitive  testing  between  alternative  approaches 
is  performed. 

The  final  activity  is  a  description  of  system  ele¬ 
ments.  Developing  as  the  result  of  previous  ac¬ 
tivities  and  as  the  final  system  design  is  deter¬ 
mined,  this  activity  takes  form  when  specifications 
are  verified  through  testing  and  when  reviewed 
in  the  Physical  Configuration  Audit  (PCA)  and 
Functional  Configuration  Audit  (FCA).  Opera¬ 
tional  testers  may  assist  in  this  activity.  They  con¬ 
duct  operational  testing  of  the  test  items/systems 
to  help  determine  the  personnel,  equipment,  fa¬ 
cilities,  software,  and  technical  data  requirements 
of  the  new  system  when  used  by  typical  military 
personnel.  Figure  2-3,  Systems  Engineering  Pro¬ 
cess  and  Test  and  Evaluation,  depicts  the  activi¬ 
ties  and  their  interactions. 

2.3.2  Technical  Management  Planning 

The  technical  management  planning  incorporates 
top-level  management  planning  for  the  integration 
of  all  system  design  activities.  Its  purpose  is  to 
develop  the  organizational  mechanisms  for  direc¬ 
tion  and  control,  and  identify  personnel  for  the 
attainment  of  cost,  performance,  and  schedule  ob¬ 
jectives.  Planning  defines  and  describes  the  type 
and  degree  of  system  engineering  management, 
the  SEP,  and  the  integration  of  related  engineer¬ 
ing  programs.  The  design  evolution  process  forms 
the  basis  for  comprehensive  T&E  planning. 

The  TEMP  must  be  consistent  with  technical 
management  planning.  The  testing  program  out¬ 
lined  in  the  TEMP  must  provide  the  TPMs  data 
required  for  all  design  decision  points,  audits,  and 
reviews  that  are  a  part  of  the  system’s  engineer¬ 
ing  process.  The  configuration  management  pro¬ 
cess  controls  the  baseline  for  the  test  programs 
and  incorporates  design  modifications  to  the 
baseline  determined  to  be  necessary  by  T&E. 
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The  TEMP  and  technical  management  planning 
must  be  traceable  to  each  other.  The  system  de¬ 
scription  in  the  TEMP  must  be  traceable  to  sys¬ 
tems  engineering  documentation  such  as  the 
FFBDs,  the  RASs,  and  the  Test  Requirements 
Sheets  (TRSs).  Key  functions  and  interfaces  of  the 
system  with  other  systems  must  be  described  and 
correlated  with  the  systems  engineering  documen¬ 
tation  and  the  system  specification.  Technical  thresh¬ 
olds  and  objectives  include  specific  performance 
requirements  that  become  test  planning  limits.  They 
must  be  traceable  through  the  planned  systems  en¬ 
gineering  documentation  and  can  be  correlated  to 
the  content  of  the  TPM  Program.  For  example,  fail¬ 
ure  criteria  for  reliability  thresholds  during  OT&E 
testing  must  be  delineated  and  agreed  upon  by  the 
Program  Manager  (PM)  and  the  Operational  Test 
Director  (OTD),  and  reflected  in  the  TEMP. 

2.3.3  Technical  Performance  Measurement 

TPM  identifies  critical  technical  parameters  that 
are  at  a  higher  level  of  risk  during  design.  It  tracks 


T&E  data,  makes  predictions  about  whether  the 
parameter  can  achieve  final  technical  success 
within  the  allocated  resources,  and  assists  in  man¬ 
aging  the  technical  program. 

The  TPM  Program  is  an  integral  part  of  the  T&E 
program.  The  TPM  is  defined  as  product  design 
assessment  and  forms  the  backbone  of  the  devel¬ 
opment  testing  program.  It  estimates,  through  engi¬ 
neering  analyses  and  tests,  the  values  of  essential 
performance  parameters  of  the  current  program 
design.  It  serves  as  a  major  input  in  the  continuous 
overall  evaluation  of  operational  effectiveness  and 
suitability.  Design  reviews  are  conducted  to  mea¬ 
sure  the  systems  engineering  progress.  For  more 
information,  see  Chapter  8.  Figure  2-4  depicts  the 
technical  reviews  that  usually  take  place  during  the 
SEP  and  the  related  specification  documents. 

2.3.4  System  Baselining  and  T&E 

The  SEP  establishes  phase  baselines  throughout 
the  acquisition  cycle.  These  baselines  (functional, 
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Figure  2-4.  Design  Reviews 


allocated,  product)  can  be  modified  with  the  re¬ 
sults  of  engineering  and  testing.  The  testing  used 
to  prove  the  technical  baselines  is  rarely  the  same 
as  the  operational  testing  of  requirements. 

Related  to  the  baseline  is  the  process  of  configu¬ 
ration  management.  Configuration  management 
benefits  the  T&E  community  in  two  ways. 
Through  configuration  management,  the  baseline 
to  be  used  for  testing  is  determined.  Also,  changes 
that  occur  to  the  baseline  as  a  result  of  testing  and 
design  reviews  are  incorporated  into  the  test  ar¬ 
ticle  before  the  new  phase  of  testing  (to  prevent 
retest  of  a  bad  design). 

2.4  DEFINITIONS 

T&E  is  the  deliberate  and  rational  generation  of 
performance  data,  which  describes  the  nature  of 
the  emerging  system  and  the  transformation  of  data 
into  information  useful  for  the  technical  and  mana¬ 
gerial  personnel  controlling  its  development.  In  the 
broad  sense,  T&E  may  be  defined  as  all  physical 
testing,  modeling,  simulation,  experimentation,  and 
related  analyses  performed  during  research,  de¬ 
velopment,  introduction,  and  employment  of  a 
weapon  system  or  subsystem.  The  Glossary  of 
Defense  Acquisition  Acronyms  and  Terms,  pro¬ 
duced  by  the  Defense  Acquisition  University  de¬ 
fines  “Test”  and  “Test  and  Evaluation”  as  follows: 

A  “test”  is  any  program  or  procedure 
which  is  designed  to  obtain,  verify,  or  pro¬ 
vide  data  for  the  evaluation  of  any  of  the 
following:  1)  progress  in  accomplishing 
developmental  objectives;  2)  the  perfor¬ 
mance,  operational  capability,  and  suitabil¬ 
ity  of  systems,  subsystems,  components, 
and  equipment  items;  and  3)  the  vulner¬ 
ability  and  lethality  of  systems,  subsystems, 
components,  and  equipment  items. 

“Test  and  Evaluation”  is  the  process  by 
which  a  system  or  components  are 
exercised  and  results  analyzed  to  provide 
performance-related  information.  The 


information  has  many  uses  including  risk 
identification  and  risk  mitigation  and  em¬ 
pirical  data  to  validate  models  and  simula¬ 
tions.  T&E  enables  an  assessment  of  the 
attainment  of  technical  performance,  speci¬ 
fications,  and  system  maturity  to  determine 
whether  systems  are  operationally  effec¬ 
tive,  suitable,  and  survivable  for  intended 
use,  and/or  lethal. 

2.5  THE  DOD  T&E  PROCESS 

The  Test  and  Evaluation  Process  (Figure  2-5)  is 
an  iterative  five  step  process  that  provides  answers 
to  critical  T&E  questions  for  decision  makers  at 
various  times  during  a  system  acquisition.  The 
T&E  process  begins  during  the  formative  stages 
of  the  program  with  the  T&E  Coordination  Func¬ 
tion,  in  which  the  information  needs  of  the  vari¬ 
ous  decision  makers  are  formulated  in  conjunc¬ 
tion  with  the  development  of  the  program 
requirements,  acquisition  strategy,  and  Analysis  of 
Alternatives  (AoAs). 

Given  certain  foundation  documentation,  Step  1 
is  the  identification  of  T&E  information  required 
by  the  decision  maker.  The  required  information 
usually  centers  on  the  current  system  under  test 
which  may  be  in  the  form  of  concepts,  prototypes, 
EDMs,  or  production  representative/production 
systems,  depending  on  the  acquisition  phase.  The 
required  information  consists  of  performance 
evaluations  of  effectiveness  and  suitability,  pro¬ 
viding  insights  into  how  well  the  system  meets 
the  use’s  needs  at  a  point  in  time. 

Step  2  is  the  pre-test  analysis  of  the  evaluation 
objectives  from  Step  1  to  determine  the  types  and 
quantities  of  data  needed,  the  results  expected  or 
anticipated  from  the  tests,  and  the  analytical  tools 
needed  to  conduct  the  tests  and  evaluations.  The 
use  of  validated  models  and  simulation  systems 
during  pre-test  analysis  can  aid  in  determining: 
how  to  design  test  scenarios;  how  to  set  up  the 
test  environment;  how  to  properly  instrument  the 
test;  how  to  staff  and  control  test  resources;  how 
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Figure  2-5.  DoDTest  and  Evaluation  Process 


best  to  sequence  the  test  trials;  and  how  to  estimate 
outcomes. 

Step  3,  test  activity  and  data  management,  is  the 
actual  test  activity  planning,  tests  are  conducted, 
and  data  management  for  data  requirements  are  iden¬ 
tified  in  Step  2.  T&E  managers  determine  what 
valid  data  exists  in  historical  files  that  can  be  ap¬ 
plied  and  what  new  data  must  be  developed  through 
testing.  The  necessary  tests  are  planned  and  ex¬ 
ecuted  to  accumulate  sufficient  data  to  support 
analysis.  Data  is  screened  for  completeness,  accu¬ 
racy,  and  validity  before  being  used  for  Step  4. 

Step  4,  post  test  synthesis  and  evaluation,  is  the 
comparison  of  the  measured  outcomes  (test  data) 
from  Step  3  with  the  expected  outcomes  from  Step 
2,  tempered  with  technical  and  operational  judg¬ 
ment.  This  is  where  data  is  synthesized  into  infor¬ 
mation.  When  the  measured  outcomes  differ  from 
the  expected  outcomes,  the  test  conditions  and 
procedures  must  be  reexamined  to  determine  if  the 


performance  deviations  are  real  or  were  the  result 
of  test  conditions,  such  as  lack  of  fidelity  in  com¬ 
puter  simulation,  insufficient  or  incorrect  test  sup¬ 
port  assets,  instrumentation  error,  or  faulty  test  pro¬ 
cesses.  The  assumptions  of  tactics,  operational 
environment,  systems  performance  parameters,  and 
logistic  support  must  have  been  carefully  chosen, 
fully  described,  and  documented  prior  to  test.  Mod¬ 
eling  and  Simulation  (M&S)  may  normally  be  used 
during  the  data  analysis  to  extend  the  evaluation  of 
performance  effectiveness  and  suitability. 

Step  5  is  when  the  decision  maker  weighs  the  T&E 
information  against  other  programmatic  informa¬ 
tion  to  decide  a  proper  course  of  action.  This  pro¬ 
cess  may  identify  additional  requirements  for  test 
data  and  iterate  the  DoD  T&E  process  again. 

2.6  SUMMARY 

T&E  is  an  engineering  tool  used  to  identify 
technical  risk  throughout  the  defense  system 
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acquisition  cycle  and  a  process  for  verifying  per¬ 
formance.  This  iterative  cycle  consists  of  acquisition 
phases  separated  by  discrete  milestones.  The  DoD 
T&E  process  consists  of  developmental  and  opera¬ 
tional  testing  that  is  used  to  support  engineering 


design  and  programmatic  reviews.  This  T&E  pro¬ 
cess  forms  an  important  part  of  the  SEP  used  by 
system  developers  and  aids  in  the  decision  process 
used  by  senior  decision  authorities  in  DoD. 
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T&E  POLICY  STRUCTURE  AND 
OVERSIGHT  MECHANISM 


3.1  INTRODUCTION 

This  chapter  provides  an  overview  of  the  policy 
and  organizations  that  govern  the  conduct  of  Test 
and  Evaluation  (T&E)  activities  within  the  De¬ 
partment  of  Defense  (DoD)  and  discusses  con¬ 
gressional  legislation  and  activities  for  compliance 
by  DoD.  It  outlines  the  responsibilities  of  DoD 
test  organizations  at  the  Office  of  the  Secretary  of 
Defense  (OSD)  and  Service  levels,  and  describes 
related  T&E  policy. 

3.2  THE  CONGRESS 

The  Congress  has  shown  a  long-standing  interest  in 
influencing  the  DoD  acquisition  process.  During  the 
early  1970s,  in  response  to  urging  by  the  Congress 
and  recommendations  by  a  Presidential  Blue  Rib¬ 
bon  Panel  on  Defense  Management,  the  Deputy 
Secretary  of  Defense,  David  Packard,  promulgated 
a  package  of  policy  initiatives  that  established  the 
Defense  Systems  Acquisition  Review  Council 
(DSARC).  The  DSARC  was  organized  to  resolve 
acquisition  issues,  whenever  possible,  and  to  pro¬ 
vide  recommendations  to  the  Secretary  of  Defense 
(SECDEF)  on  the  acquisition  of  major  weapon  sys¬ 
tems.  Also,  as  a  result  of  the  Congressional  Direc¬ 
tives,  the  Army  and  Air  Force  established  indepen¬ 
dent  Operational  Test  Agencies  (OTAs).  The  Navy 
Operational  Test  and  Evaluation  Force  was  estab¬ 
lished  in  the  late  1960s.  In  1983,  similar  concerns 
led  the  Congress  to  direct  the  establishment  of  the 
independent  Office  of  the  Director,  Operational  Test 
and  Evaluation  (DOT&E),  within  OSD.  In  1985  a 
report  released  by  another  President’s  Blue  Ribbon 
Commission  on  Defense  Management,  this  time 
chaired  by  David  Packard,  made  significant  recom¬ 
mendations  on  the  management  and  oversight  of 


DoD’s  acquisition  process,  specifically,  T&E.  All  the 
Commission’s  recommendations  have  not  been 
implemented,  and  the  full  impact  of  these  recom¬ 
mendations  is  not  yet  realized.  In  Fiscal  Year  (FY)87 
the  Defense  Authorization  Act  required  Live  Fire 
Testing  (LFT)  of  weapon  systems  before  the  Pro¬ 
duction  Phase  begins.  The  earmarking  of  authoriza¬ 
tions  and  appropriations  for  DoD  funding,  specific 
testing  requirements,  and  acquisition  reform  legisla¬ 
tion  continues  to  indicate  the  will  of  the  Congress 
for  DoD  implementation. 

Congress  requires  DoD  to  provide  the  following 
reports  that  include  information  on  T&E: 

•  Selected  Acquisition  Report  (SAR).  Within  the 
cost,  schedule,  and  performance  data  in  the  re¬ 
port,  SAR  describes  Acquisition  Category 
(ACAT)  I  system  characteristics  required  and 
outlines  significant  progress  and  problems  en¬ 
countered.  It  lists  tests  completed  and  issues 
identified  during  testing.  The  Program  Manager 
(PM)  uses  the  Consolidated  Acquisition  Report¬ 
ing  System  (CARS)  software  to  prepare  the 
SAR. 

•  Annual  System  Operational  Test  Report.  This 
report  is  provided  by  the  DOT&E  to  the  SEC¬ 
DEF  and  the  committees  on  Armed  Services, 
National  Security,  and  Appropriations.  The  re¬ 
port  provides  a  narrative  and  resource  summary 
of  all  Operational  Test  and  Evaluation  (OT&E) 
and  related  issues,  activities,  and  assessments. 
When  oversight  of  LFT  was  moved  to 
DOT&E,  this  issue  was  added  to  the  report. 

•  Beyond  Low  Rate  Initial  Production  (BLRIP) 
Report.  Before  proceeding  BLRIP  for  each 


Major  Defense  Acquisition  Program  (MDAP), 
DOT&E  must  report  to  the  SECDEF  and  the 
Congress.  This  report  addresses  the  adequacy 
of  OT&E  and  whether  the  T&E  results  con¬ 
firm  that  the  tested  item  or  component  is  effec¬ 
tive  and  suitable  for  combat.  When  oversight 
of  LFT  was  moved  to  the  DOT&E,  the  Live 
Fire  Test  Report  was  added  to  the  BLRIP  report 
content. 

•  Foreign  Comparative  Test  (FCT)  Report.  The 
Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (USD(AT&L)) 
should  notify  the  Congress  a  minimum  of  30 
days  prior  to  the  commitment  of  funds  for  initia¬ 
tion  of  new  FCT  evaluations  of  equipment  pro¬ 
duced  by  selected  allied  and  friendly  foreign 
countries. 

3.3  OSD  OVERSIGHT  STRUCTURE 

The  DoD  organization  for  the  oversight  of  T&E 
is  illustrated  in  Figure  3-1.  In  USD(AT&L), 
DT&E  oversight  is  performed  by  the  Deputy  Di¬ 
rector,  Developmental  Test  and  Evaluation,  De¬ 
fense  Systems  (DT&E/DS)  and  the  DOT&E  pro¬ 
vides  OT&E  oversight  for  the  SECDEF.  The 
management  of  MDAPs  in  OSD  is  performed  by 
the  Defense  Acquisition  Executive  (DAE),  who 
uses  the  Defense  Acquisition  Board  (DAB)  and 
Overarching  Integrated  Product  Teams  (OIPTs) 
to  process  information  for  decisions. 

3.3.1  Defense  Acquisition  Executive  (DAE) 

The  DAE  position,  established  in  September 
1986,  is  held  by  the  USD(AT&L).  As  the  DAE, 
the  USD(AT&L)  uses  the  DAB  and  its  OIPTs 
to  provide  the  senior-level  decision  process  for 
the  acquisition  of  weapon  systems.  The  DAE  re¬ 
sponsibilities  include  establishing  policies  for  ac¬ 
quisition  (including  procurement,  Research  and 
Development  (R&D),  logistics,  development  test¬ 
ing,  and  contracts  administration)  for  all  elements 
of  DoD.  His  charter  includes  the  authority  over 
the  Service  and  defense  agencies  on  policy. 


procedure,  and  execution  of  the  weapon  systems 
acquisition  process. 

3.3.2  Defense  Acquisition  Board  (DAB) 

The  DAB  is  the  primary  forum  used  by  OSD  to 
provide  advice,  assistance,  and  recommendations, 
and  to  resolve  issues  regarding  all  operating  and 
policy  aspects  of  the  DoD  acquisition  system.  The 
DAB  is  the  senior  management  acquisition  board 
chaired  by  the  DAE  and  Co-chair  is  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  (VCJCS). 
The  DAB  is  composed  of  the  department’s  senior 
acquisition  officials,  including  the  DOT&E.  The 
DAB  conducts  business  through  OIPTs  and  pro¬ 
vides  decisions  on  ACAT  ID  programs  (DoD  In¬ 
struction  (DoDI)  5000.2,  Operation  of  the  Defense 
Acquisition  System ,  12  May  2003). 

3.3.3  Deputy  Director,  Developmental  Test 
and  Evaluation,  Defense  Systems 
(DT&E/DS) 

The  DT&E/DS  serves  as  the  principal  staff  assis¬ 
tant  and  advisor  to  the  USD(AT&L)  for  T&E 
matters.  The  Director,  Defense  Systems  works  for 
the  Deputy  Under  Secretary  of  Defense  for  Ac¬ 
quisition  and  Technology  (DUSD(A&T))  and  has 
authority  and  responsibility  for  all  Development 
Test  and  Evaluation  (DT&E)  conducted  on  des¬ 
ignated  MDAPs  and  for  FCT.  The  DT&E/DS  is 
illustrated  in  Figure  3-1. 

3.3.3.1  Duties  of  the  DT&E/DS 

Within  the  acquisition  community,  the  DT&E/DS: 

•  Serves  as  the  focal  point  for  coordination  of  all 
MDAP  Test  and  Evaluation  Master  Plans 
(TEMPs).  Recommends  approval  by  the  OIPT 
lead  of  the  DT&E  portion  of  TEMPs; 

•  Reviews  MDAP  documentation  for  DT&E  im¬ 
plications  and  resource  requirements  to  provide 
comments  to  the  OIPT,  USD(AT&L)  DAE  or 
DAB; 
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Figure  3-1.  DoD  Test  and  Evaluation  Organization 


•  Monitors  DT&E  to  ensure  the  adequacy  of  test¬ 
ing  and  to  assess  test  results; 

•  Provides  assessments  of  the  Defense  Acquisi¬ 
tion  Executive  Summary  (DAES)  and  techni¬ 
cal  assessments  of  DT&E  and  system  engineer¬ 
ing  processes  conducted  on  a  weapon  system; 

•  Provides  advice  and  makes  recommendations 
to  OSD  senior  leadership,  and  issues  guidance 
to  the  Component  Acquisition  Executives 
(CAEs)  with  respect  to  DT&E; 

•  Leads  the  DS  efforts  for  joint  testing  and  is  a 
member  of  the  Joint  T&E  Program’s  Technical 
Advisory  Board  for  Joint  Development  Test  and 
Evaluation  programs; 

3.3.3.2  DT&E/DS  and  Service  Reports 

During  the  testing  of  ACAT  I  and  designated 
weapon  systems,  the  DT&E/DS  and  Services  in¬ 
teraction  includes  the  following  reporting  require¬ 
ments: 

•  A  TEMP  (either  preliminary  or  updated,  as 
appropriate)  must  be  provided  for  consideration 
and  approval  before  each  milestone  review, 
starting  with  Milestone  (MS)  B. 

•  A  technical  assessment  of  Developmental  T&E 
is  provided  to  the  DS  and  DOT&E  listing  the 
T&E  results,  conclusions,  and  recommenda¬ 
tions  prior  to  a  milestone  decision  or  the  final 
decision  to  proceed  BLRIP. 

3.3.4  Director,  Operational  Test  and 
Evaluation  (DOT&E) 

As  illustrated  in  Figure  3-1,  the  director  reports 
directly  to  the  SECDEF  and  has  special  reporting 
requirements  to  the  Congress.  The  DOT&E’s  re¬ 
sponsibility  to  the  Congress  is  to  provide  an  unbi¬ 
ased  window  of  insight  into  the  operational  effec¬ 
tiveness,  suitability,  and  survivability  of  new 
weapon  systems. 


3.3.4.1  Duties  and  Functions  of  the  DOT&E 

The  specific  duties  of  DOT&E  are  outlined  in 

DoD  Directive  5141.2,  Director  of  Operational 

Test  and  Evaluation  (DOT&E),  25  May  2000 

(Reference  13). 

The  functions  of  the  office  include: 

•  Obtaining  reports,  information,  advice,  and  as¬ 
sistance  as  necessary  to  carry  out  assigned  func¬ 
tions  DOT&E  has  access  to  all  records  and  data 
in  DoD  on  acquisition  programs); 

•  Signing  the  ACAT  I  and  oversight  TEMPs  for 
approval  of  OT&E  and  approving  the  OT&E 
funding  for  major  systems  acquisition; 

•  Approving  operational  test  plans  on  all  major 
defense  acquisition  systems  and  designated 
oversight  programs  prior  to  system  starting  ini¬ 
tial  operational  testing  (approval  in  writing  re¬ 
quired  before  operational  testing  may  begin) 
oversight  extends  into  Follow-on  Operational 
Test  and  Evaluation  (FOT&E); 

•  Providing  observers  during  preparation  and 
conduct  of  OT&E; 

•  Analyzing  results  of  OT&E  conducted  for  each 
major  or  designated  defense  acquisition  pro¬ 
gram  and  submitting  a  report  to  the  SECDEF 
and  the  Congress  on  the  adequacy  of  the  OT&E 
performed; 

•  A  final  decision  to  proceed  with  a  major  pro¬ 
gram  BLRIP  cannot  be  made  until  DOT&E 
has  reported  (BLRIP  Report)  to  the  SECDEF 
and  to  congressional  Committees  on  Armed 
Services  and  Appropriations  on  the  adequacy 
of  LFT&E  and  OT&E  and  whether  the  results 
confirm  the  system’s  operational  effectiveness 
and  suitability; 

•  Provide  oversight  and  approval  of  major 
program  LFT. 
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3.3.4.2  DOT&E  and  Service  Interactions 

For  DoD  and  DOT&E-designated  oversight  ac¬ 
quisition  programs,  the  Service  provides  the 

DOT&E  the  following: 

•  A  draft  copy  of  the  Operational  Test  Plan  con¬ 
cept  for  review; 

•  Significant  Test  Plan  changes; 

•  The  final  Service  IOT&E  report  which  must 
be  submitted  to  DOT&E  before  the  Full  Rate 
Production  Decision  Review  (FRPDR); 

•  The  Live  Fire  Test  and  Evaluation  (LFT&E) 
plan  for  approval  and  the  Service  Live  Fire  Test 
Report  for  review. 


3.4  SERVICE  T&E  MANAGEMENT 
STRUCTURES 

3.4.1  Army  T&E  Organizational 
Relationship 

The  Army  management  structure  for  T&E  is  il¬ 
lustrated  in  Figure  3-2. 

3.4.1.1  Army  Acquisition  Executive  (AAE) 

The  Assistant  Secretary  of  the  Army  for  Acquisi¬ 
tion,  Logistics,  and  Technology  (ASA(ALT))  has 
the  principal  responsibility  for  all  Department  of 
the  Army  (DA)  matters  and  policy  related  to  ac¬ 
quisition,  logistics,  technology,  procurement,  the 
industrial  base,  and  security  cooperation.  Addition¬ 
ally,  the  ASA(ALT)  serves  as  the  Army  Acquisi¬ 
tion  Executive  (AAE).  The  AAE  administers  ac¬ 
quisition  programs  by  developing/promulgating 
acquisition  policies  and  procedures  as  well  as 
appointing,  supervising,  and  evaluating  assigned 


Figure  3-2.  ArmyT&E  Organization 
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Program  Executive  Officers  (PEOs)  and  direct 
reporting  PMs. 

3.4.1.2  Army  T&E  Executive 

The  Deputy  Under  Secretary  of  the  Army  for 
Operations  Research  (DUSA(OR)),  establishes, 
reviews,  enforces,  and  supervises  Army  T&E 
policy  and  procedures  including  overseeing  all 
Army  T&E  associated  with  the  system  research, 
development,  and  acquisition  of  all  materiel  and 
Command,  Control,  Communications,  and  Com¬ 
puters  (C4)  and  Intelligence  (C4I)/Information 
Technology  (IT)  systems.  The  Test  and  Evalua¬ 
tion  Management  Agency  (TEMA)  within  the 
Office  of  the  Chief  of  Staff,  Army,  provides  the 
DUSA(OR)  oversight.  As  delegated  by  the  AAE, 
the  DUSA(OR)  is  the  sole  Headquarters,  Depart¬ 
ment  of  the  Army  (HQDA)  approval  authority  for 
TEMPs. 

3.4.1.3  Army  Test  and  Evaluation  Command 

The  U.S.  Army  Test  and  Evaluation  Command 
(ATEC)  supports  the  systems  acquisition,  force  de¬ 
velopment,  and  experimentation  processes  through 
overall  management  of  the  Army’s  T&E  programs. 
In  this  role,  ATEC  manages  the  Army’s  develop¬ 
mental  and  operational  testing,  all  system  evalua¬ 
tion,  as  well  as  the  management  of  joint  T&E. 
ATEC  is  the  Army’s  independent  Operational  Test 
Activity  (OTA)  reporting  directly  to  the  Vice  Chief 
of  Staff,  Army  (VCSA)  through  the  Director  of 
the  Army  Staff  (DAS). 

3.4.1.3.1  Army  Developmental  Testers 

The  U.S.  Army  Developmental  Test  Command 
(DTC)  within  ATEC  has  the  primary  responsibil¬ 
ity  for  conducting  Developmental  Tests  (DTs)  for 
the  Army.  DTC  responsibilities  include: 

•  Perform  the  duties  of  governmental  developmen¬ 
tal  tester  for  all  Army  systems  except  for  those 
systems  assigned  to  the  Communications- 
Electronics  Research,  Development,  and 


Engineering  Center  (CERDEC)  of  the  Army 
Materiel  Command  (AMC)  Research  Develop¬ 
ment  and  Engineering  Command  (RDECOM) 
by  HQDA  (Deputy  Chief  of  Staff,  G-6),  Medi¬ 
cal  Command  (MEDCOM),  Intelligence  and  Se¬ 
curity  Command  (INSCOM),  Space  and  Mis¬ 
sile  Defense  Command  (SMDC),  and  Army 
Corps  of  Engineers  (ACE); 

•  Provide  test  facilities  and  testing  expertise  in 
support  of  the  acquisition  of  Army  and  other 
defense  systems; 

•  Operate  and  maintain  the  Army’s  portion  of  the 
Major  Range  and  Test  Facility  Base  (MRTFB) 
(except  for  the  United  States  Army  Kwajalein 
Atoll  (USAKA)  and  the  High  Energy  Laser 
System  Test  Facility  (HELSTF))  in  accordance 
with  DoD  Directive  (DoDD)  3200.11,  Major 
Range  and  Test  Facility  Base  (MRTFB),  1  May 
2002; 

•  Provide  testers  with  a  safety  release  for  sys¬ 
tems  before  the  start  of  pretest  training  for  tests 
that  use  soldiers  as  test  participants; 

•  Provide  safety  confirmations  for  milestone 
acquisition  decision  reviews  and  the  materiel 
release  decision; 

•  Manage  the  Army  Test  Incident  Reporting 
System  (ATIRS). 

3.4.1.3.2  Army  Operational  Testers 

The  U.S.  Army  Operational  Test  Command  (OTC) 

within  ATEC  has  the  primary  responsibility  for 

conducting  Operational  Tests  (OTs)  for  the  Army 

and  supporting  Army  participation  in  Joint  T&E. 

OTC  responsibilities  include: 

•  Perform  the  duties  of  operational  tester  for  all 
Army  systems  except  for  those  systems  as¬ 
signed  to  MEDCOM,  INSCOM,  SMDC,  and 
ACE; 
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•  Perform  the  duties  of  operational  tester  for  as¬ 
signed  multi-Service  OTs  and  (on  a  customer 
service  basis)  for  Training  and  Doctrine  Com¬ 
mand  (TRADOC)  Force  Development  Tests 
and  Experimentation  (FDTE); 

•  Provide  test  facilities  and  testing  expertise  in 
support  of  the  acquisition  of  Army  and  other 
defense  systems,  and  for  other  customers  on  a 
cost  reimbursable  and  as  available  basis; 

•  Program  and  budget  the  funds  to  support  OT 
except  out  of  cycle  tests  (which  are  usually  paid 
for  by  the  proponent); 

•  Develop  and  submit  OT  and  FDTE  Outline 
Test  Plans  (OTPs)  to  the  Army’s  Test  Sched¬ 
ule  and  Review  Committee  (TSARC). 

3.4.1.3.3  Army  Evaluation  Center 

The  U.S.  Army  Evaluation  Center  (AEC)  is  an 
independent  subordinate  element  within  ATEC  that 
has  the  primary  responsibility  for  conducting  Army 
system  evaluations  and  system  assessments  in  sup¬ 
port  of  the  systems  acquisition  process.  Decision 
makers  use  AEC’s  independent  report  addressing 
an  Army  system’s  operational  effectiveness,  suitabil¬ 
ity,  and  survivability.  AEC  responsibilities  include: 

•  Perform  the  duties  of  system  evaluator  for  all 
Army  systems  except  for  those  systems  assigned 
to  MEDCOM,  INSCOM,  and  the  commercial 
items  assigned  to  ACE;  conduct  Continuous 
Evaluation  (CE)  on  all  assigned  systems; 

•  Develop  and  promulgate  evaluation  capabili¬ 
ties  and  methodologies; 

•  Coordinate  system  evaluation  resources  through 
the  TSARC; 

•  Preview  programmed  system  evaluation  require¬ 
ments  for  possible  use  of  Modeling  and  Simula¬ 
tion  (M&S)  to  enhance  evaluation  and  reduce 
costs;  perform  Manpower  and  Personnel 


Integration  (MANPRINT)  assessments  in  co¬ 
ordination  with  Deputy  Chief  of  Staff  (DCS), 
G-l  Army  Research  Laboratory-Human  Re¬ 
search  and  Engineering  Division/Directorate 
(ARL-HRED); 

Perform  the  Integrated  Logistics  Support  (ILS) 
program  surveillance  for  Army  systems.  Perform 
independent  logistics  supportability  assessments 
and  report  them  to  the  Army  Logistician  and  other 
interested  members  of  the  acquisition  community. 

3.4.2  Navy  T&E  Organizational  Relationship 

The  organizational  structure  for  T&E  in  the  Navy  is 
illustrated  in  Figure  3-3.  Within  the  Navy  Secretariat, 
the  Secretary  of  the  Navy  (SECNAV)  has  assigned 
general  and  specific  Research,  Development,  Test, 
and  Evaluation  (RDT&E)  responsibilities  to  the  As¬ 
sistant  Secretary  of  the  Navy  (Research,  Develop¬ 
ment,  and  Acquisition)  (ASN(RDA))  and  to  the 
Chief  of  Naval  Operations  (CNO).  The  CNO  has 
responsibility  for  ensuring  the  adequacy  of  the 
Navy’s  overall  T&E  program.  The  T&E  policy  and 
guidance  are  exercised  through  the  Director  of  Navy 
T&E  and  Technology  Requirements  (N-91).  Staff 
support  is  provided  by  the  Test  and  Evaluation  Di¬ 
vision  (N-91 2)  which  has  cognizance  over  planning, 
conducting,  and  reporting  all  T&E  associated  with 
development  of  systems. 

3.4.2.2  Navy  DT&E  Organizations 

The  Navy’s  senior  systems  development  authority 
is  divided  among  the  commanders  of  the  system 
commands  with  Naval  Air  Systems  Command 
(NAVAIR)  developing  and  performing  DT&E  on 
aircraft  and  their  essential  weapon  systems;  Naval 
Sea  Systems  Command  (NAVSEA)  developing  and 
performing  DT&E  on  ships,  submarines,  and  their 
associated  weapon  systems  and  Space  and  Naval 
Warfare  Systems  Command  (SPAWAR)  develop¬ 
ing  and  performing  DT&E  on  all  other  systems. 
System  acquisition  is  controlled  by  a  chartered  PM 
or  by  the  commander  of  a  systems  command.  In 
both  cases,  the  designated  developing  agency  is 
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Figure  3-3.  NavyT&E  Organization 


responsible  for  DT&E  and  for  the  coordination  of 
all  T&E  planning  in  the  TEMP.  Developing  Agen¬ 
cies  (DAs)  are  responsible  for: 

•  Developing  test  issues  based  on  the  thresholds 
established  by  the  user  in  the  requirements 
documents; 

•  Identifying  the  testing  facilities  and  resources 
required  to  conduct  the  DT&E; 

•  Developing  the  DT&E  test  reports  and  quick- 
look  reports. 

3.4.2.3  Navy  Operational  Test  and 
Evaluation  Force 

The  Commander,  Operational  Test  and  Evalua¬ 
tion  Force  (COMOPTEVFOR),  commands  the 
Navy’s  independent  OT&E  activity  and  reports 


directly  to  the  CNO.  The  functions  of  the 

COMOPTEVFOR  include: 

•  Establishing  early  liaison  with  the  DA  to  en¬ 
sure  an  understanding  of  the  test  requirements 
and  plans; 

•  Reviewing  acquisition  program  documentation 
to  ensure  that  documents  are  adequate  to  sup¬ 
port  a  meaningful  T&E  program; 

•  Planning  and  conducting  realistic  OT&E; 

•  Developing  tactics  and  procedures  for  the  em¬ 
ployment  of  systems  that  undergo  OT&E  (as 
directed  by  the  CNO); 

•  Providing  recommendations  to  the  CNO  for  the 
development  of  new  capabilities  or  the  upgrade 
of  ranges; 
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•  Also  reporting  directly  to  the  CNO,  the  Presi¬ 
dent  of  the  Board  of  Inspection  and  Survey 
(PRESINSURV)  is  responsible  for  conducting 
acceptance  trials  of  new  ships  and  aircraft  ac¬ 
quisitions  and  is  the  primary  Navy  authority 
for  Production  Acceptance  Test  and  Evaluation 
(PAT&E)  of  these  systems; 

•  Conducting  OT&E  on  aviation  systems  in  con¬ 
junction  with  Marine  Corps  Operational  Test 
and  Evaluation  Activity  (MCOTEA). 

3.4.3  Air  Force  Organizational  Relationships 

3.4.3.1  Air  Force  Acquisition  Executive 

The  Assistant  Secretary  of  the  Air  Force  for  Ac¬ 
quisition  (SAF/AQ)  is  the  senior-level  authority 
for  Research,  Development,  and  Acquisition 
(RD&A)  within  the  Air  Force.  As  illustrated  in 
Figure  3-4,  the  SAF/AQ  is  an  advisor  to  the  Sec¬ 
retary  of  the  Air  Force  and  interfaces  directly  with 
the  DT&E  and  DOT&E.  The  SAF/AQ  receives 


DT&E  and  OT&E  results  as  a  part  of  the  acquisi¬ 
tion  decision  process.  Within  the  SAF/AQ  struc¬ 
ture,  there  is  a  military  deputy  (acquisition)  who  is 
the  Air  Force  primary  staff  officer  with  responsi¬ 
bility  for  RD&A.  This  staff  officer  is  the  chief  ad¬ 
vocate  of  Air  Force  acquisition  programs  and  de¬ 
velops  the  RDT&E  budget.  Air  Force  policy  and 
oversight  for  T&E  is  provided  by  a  staff  element 
under  the  Chief  of  Staff,  Test  and  Evaluation  (AF/ 
TE).  They  process  test  documentation  for  DT&E 
and  OT&E  and  manage  the  review  of  the  TEMP. 

3.4.3.2  Air  Force  DT&E  Organization 

The  Air  Force  Materiel  Command  (AFMC)  is  the 
primary  DT&E  and  acquisition  manager.  The 
AFMC  performs  all  levels  of  research;  develops 
weapon  systems,  support  systems,  and  equipment; 
and  conducts  all  DT&E.  The  acquisition  PMs  are 
under  the  Commander,  AFMC.  Within  the  AFMC, 
there  are  major  product  divisions,  test  centers,  and 
laboratories  as  well  as  missile,  aircraft,  and 
munitions  test  ranges. 


Figure  3-4.  Air  Force  T&E  Organization 
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Once  the  weapon  system  is  fielded,  AFMC  re¬ 
tains  management  responsibility  for  developing 
and  testing  system  improvements,  enhancements, 
or  upgrades. 

3.4.3.3  Air  Force  OT&E  Organization 

The  AF/TE  is  responsible  for  supporting  and  coor¬ 
dinating  the  OT&E  activities  of  the  Air  Force  Op¬ 
erational  Test  and  Evaluation  Center  (AFOTEC). 

The  Commander,  AFOTEC,  is  responsible  to  the 
Secretary  of  the  Air  Force  and  the  Chief  of  Staff 
for  the  independent  T&E  of  all  major  and  non¬ 
major  systems  acquisitions.  The  Commander  is 
supported  by  the  operational  commands  and  oth¬ 
ers  in  planning  and  conducting  OT&E. 

The  AFOTEC  reviews  operational  requirements, 
employment  concepts,  tactics,  maintenance 
concepts,  and  training  requirements  before  con¬ 
ducting  OT&E.  The  operational  commands  pro¬ 
vide  operational  concepts,  personnel,  and  re¬ 
sources  to  assist  AFOTEC  in  performing  OT&E. 

3.4.4  Marine  Corps  Organizational 
Relationship 

3.4.4. 1  Marine  Corps  Acquisition  Executive 

The  Deputy  Chief  of  Staff  for  Research  and  De¬ 
velopment  (DCS/R&D),  Headquarters  Marine 
Corps,  directs  the  total  Marine  Corps  RDT&E  ef¬ 
fort  to  support  the  acquisition  of  new  systems.  The 
DCS/R&D’s  position  within  the  General  Staff  is 
analogous  to  that  of  the  Director,  T&E,  Tech/N-91 
in  the  Navy  structure.  The  DCS/R&D  also  reports 
directly  to  the  Assistant  Secretary  of  the  Navy/ 
Research,  Engineering,  and  Science  (ASN/RE&S) 
in  the  Navy  Secretariat.  Figure  3-3,  illustrates  the 
Marine  Corps  organization  for  T&E  management. 

3.4.4.2  Marine  Corps  DT&E  Organizations 

The  Commanding  General,  Marine  Corps  Systems 
Command  (CG  MCSC),  is  the  Marine  Corps 


materiel  developing  agent  and  directly  interfaces 
with  the  Navy  Systems  Commands.  The  CG 
MCSC  implements  policies,  procedures,  and  re¬ 
quirements  for  DT&E  of  all  systems  acquired  by 
the  Marine  Corps.  The  Marine  Corps  also  uses 
DT&E  and  OT&E  performed  by  other  Services, 
which  may  develop  systems  of  interest  to  the 
Corps. 

3.4.4.3  Marine  Corps  Operational  Test  and 
Evaluation  Activity  (MCOTEA) 

The  MCOTEA  is  the  independent  OT&E  activity 
maintained  by  the  Marine  Corps.  Its  function  is 
analogous  to  that  performed  by  Operational  Test 
and  Evaluation  Force  (Navy)  (OPTEYFOR)  in 
the  Navy.  The  CG  MCSC  provides  direct  assis¬ 
tance  to  MCOTEA  in  the  planning,  conducting, 
and  reporting  of  OT&E.  MC  aviation  systems  are 
tested  by  OPTEVFOR.  The  Fleet  Marine  Force 
performs  troop  T&E  of  materiel  development  in 
an  operational  environment. 

3.5  THE  T&E  EXECUTIVE  AGENT 
STRUCTURE 

In  1993  the  USD(AT&L)  approved  a  T&E  Ex¬ 
ecutive  Agent  structure  to  provide  the  Services 
with  more  corporate  responsibility  for  the  man¬ 
agement  and  policies  that  influence  the  availabil¬ 
ity  of  test  resources  for  the  evaluation  of  DoD 
systems  in  acquisition  (Figure  3-5).  The  DT&E/ 
DS  has  functional  responsibility  for  the  execution 
of  the  processes  necessary  to  assure  the  T&E 
Executive  Agent  structure  functions  effectively. 
The  DT&E/DS  also  participates  in  the  Operational 
Test  and  Evaluation  Coordinating  Committee, 
chaired  by  the  Director,  Operational  Test  and 
Evaluation  (DOT&E).  This  committee  manages 
the  OT&E  Resources  Enhancement  Project  and 
the  DT&E/DS  draws  input  to  the  T&E  Executive 
Agent  structure  for  coordination  of  all  T&E  re¬ 
source  requirements. 

The  Board  of  Directors  (BoD)  (Service  Vice 
Chiefs)  is  assisted  by  an  Executive  Secretariat 
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Figure  3-5.  T&E  Executive  Agent  Structure 


consisting  of  the  Army  DUSA(OR),  the  Navy  N- 
91,  and  the  Air  Force  AF/TE.  The  BoD  provides 
guidance  and  decisions  on  policy  and  resource  al¬ 
location  to  their  subordinate  element,  the  Board 
of  Operating  Directors  ((BoOD)  (Army  DTC, 
NAVAIR  5.0,  and  AFMC  DO).  The  BoD  also 
provides  program  review  and  advocacy  support 
of  the  T&E  infrastructure  to  OSD  and  Congress. 

The  BoOD  is  supported  by  a  Secretariat  and  the 
Defense  Test  and  Training  Steering  Group 
(DTTSG).  The  DTTSG  manages  the  T&E  Re¬ 
sources  Committee  (TERC),  the  Training  Instru¬ 
mentation  Resource  Investment  Committee 
(TIRIC),  and  the  CROSSBOW  Committee.  The 
DTTSG  is  instrumental  in  achieving  efficient  ac¬ 
quisition  and  integration  of  all  training  and  asso¬ 
ciated  test  range  instrumentation  and  the  develop¬ 
ment  of  acquisition  policy  for  embedded  weapon 
system  training  and  testing  capabilities.  The  TERC 


supports  the  DTTSG  in  overseeing  infrastructure 
requirements  development  from  a  T&E  commu¬ 
nity  perspective,  both  development  testing  and 
operational  testing,  and  manages  OSD  funding  the 
execution  of  the  Central  T&E  Investment  Program 
(CTEIP).  The  TIRIC  is  chartered  to  ensure  the 
efficient  acquisition  of  common  and  interoperable 
range  instrumentation  systems.  The  CROSSBOW 
Committee  provides  technical  and  management 
oversight  of  the  Services’  development  and  ac¬ 
quisition  programs  for  threat  and  threat  related 
hardware  simulators,  emitters,  software  simula¬ 
tions,  hybrid  representations,  and  surrogates. 

3.6  SUMMARY 

An  increased  emphasis  on  T&E  has  placed  greater 
demands  on  the  OSD  and  DoD  components  to 
carefully  structure  organizations  and  resources  to 
ensure  maximum  effectiveness.  Renewed  interest 
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by  Congress  in  testing  as  a  way  of  assessing  Services.  These  policy  changes  and  reorganiza- 
systems  utility  and  effectiveness,  the  report  by  the  tions  will  be  ongoing  for  several  years  to  improve 

President’s  Blue  Ribbon  Panel  on  Acquisition  the  management  of  T&E  resources  in  support  of 

Management,  and  acquisition  reform  initiatives  acquisition  programs, 
have  resulted  in  major  reorganizations  within  the 
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PROGRAM  OFFICE  RESPONSIBILITIES 
FOR  TEST  AND  EVALUATION 


4.1  INTRODUCTION 

In  government  acquisition  programs,  there  should 
be  an  element  dedicated  to  management  of  Test 
and  Evaluation  (T&E).  This  element  would  have 
the  overall  test  program  responsibility  for  all  phases 
of  the  acquisition  process.  T&E  expertise  may  be 
available  through  matrix  support  or  reside  in  the 
Program  Management  Office  (PMO)  engineering 
department  during  the  program’s  early  phases.  By 
System  Demonstration,  the  PMO  should  have  a 
dedicated  T&E  manager.  In  the  PMO,  the  Deputy 
for  T&E  would  be  responsible  for  defining  the 
scope  and  concept  of  the  test  program,  establish¬ 
ing  the  overall  program  test  objectives  and  man¬ 
aging  test  program  funds  and  coordination.  The 
Deputy  for  T&E  should  provide  test  directors 
(such  as  a  joint  test  director)  as  required,  and  co¬ 
ordinate  the  test  resources,  facilities  and  their  sup¬ 
port  required  for  each  phase  of  testing.  In  addi¬ 
tion,  the  Deputy  for  T&E  or  a  staff  member,  will 
be  responsible  for  managing  the  Test  and  Evalua¬ 
tion  Master  Plan  (TEMP)  and  planning  and  man¬ 
aging  any  special  test  programs  required  for  the 
program.  The  Deputy  for  T&E  will  also  review, 
evaluate,  approve,  and  release  for  distribution 
contractor-prepared  test  plans  and  reports,  and  re¬ 
view  and  coordinate  all  appropriate  government 
test  plans.  After  the  system  is  produced,  the 
Deputy  for  T&E  will  be  responsible  for  support¬ 
ing  production  acceptance  testing  and  the  test  por¬ 
tions  of  later  increments,  Preplanned  Product  Im¬ 
provements  (P3I)  upgrades  or  enhancements  to  the 
weapon  system/acquisition.  If  the  program  is  large 
enough,  the  Deputy  for  T&E  will  be  responsible 
for  all  T&E  direction  and  guidance  for  that 
program. 


4.2  RELATIONSHIP  TO  THE 
PROGRAM  MANAGER 

The  Program  Manager  (PM)  is  ultimately  respon¬ 
sible  for  all  aspects  of  the  system  development, 
including  testing.  The  Deputy  for  T&E  is  normally 
authorized  by  the  PM  to  conduct  all  duties  in  the 
area  of  T&E.  The  input  of  the  Deputy  for  T&E 
to  the  contract,  engineering  specifications,  bud¬ 
get,  program  schedule,  etc.,  is  essential  for  the  PM 
to  manage  the  program  efficiently. 

4.3  EARLY  PROGRAM  STAGES 

In  the  early  stages  of  the  program,  the  T&E  func¬ 
tion  is  often  handled  by  matrix  support  from  the 
material  command.  Matrix  T&E  support  or  the 
Deputy  for  T&E  should  be  responsible  for  devel¬ 
opment  of  the  T&E  sections  of  the  Request  for 
Proposal  (RFP).  Although  the  ultimate  responsi¬ 
bility  for  the  RFP  is  between  the  PM  and  the  Pri¬ 
mary  Contracting  Officer  (PCO),  the  Deputy  for 
T&E  is  responsible  for  creating  several  sections. 
These  sections  include  the  test  schedule,  test  pro¬ 
gram  funding  (projections),  test  data  requirements 
for  the  program  (test  reports,  plans,  procedures, 
quick-look  reports,  etc.),  the  test  section  of  the  State¬ 
ment  of  Work  (SOW),  portions  of  the  Acquisition 
Plan,  Information  for  Proposal  Preparation  (IFPP), 
and  (if  a  joint  acquisition  program)  feedback  on 
the  Capability  Development  Document  (CDD)  in¬ 
tegrating  multiple  Service’s  requirements. 

4.3.1  Memorandums 

Early  in  the  program,  another  task  of  the  Deputy 
for  T&E  is  the  arrangement  of  any  Memorandums 
of  Agreement  or  Understanding  (MOA/MOU) 
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between  Services,  North  Atlantic  Treaty  Organi¬ 
zation  (NATO)  countries,  test  organizations,  etc., 
which  outline  the  responsibilities  of  each  organi¬ 
zation.  The  RFP  and  SOW  outline  contractor/ 
government  obligations  and  arrangements  on  the 
access  and  use  of  test  facilities  (contractor  or  gov¬ 
ernment  owned). 

4.3.2  Test  Data  Management 

The  Deputy  for  T&E  may  have  approval  author¬ 
ity  for  all  contractor-created  test  plans,  procedures, 
and  reports.  The  Deputy  for  T&E  must  have  ac¬ 
cess  to  all  contractor  testing  and  test  results,  and 
the  Deputy  for  T&E  is  responsible  for  dissemi¬ 
nating  the  results  to  government  agencies  that  need 
this  data.  Additionally,  the  Deputy  for  T&E  cre¬ 
ates  report  formats  and  time  lines  for  contractor 
submittal,  government  approval,  etc. 

The  data  requirements  for  the  entire  test  program 
are  outlined  in  the  Contract  Data  Requirements  List 
(CDRL).  The  Deputy  for  T&E  should  review  the 
Acquisition  Management  Systems  and  Data  Re¬ 
quirements  Control  List  (AMSDL),  Department  of 
Defense  (DoD)  5010. 12-L,  April  2001 ,  for  relevant 
test  Data  Item  Descriptions  (DIDs).  (Examples  can 
be  found  in  Appendix  C.)  The  Deputy  for  T&E 
provides  input  to  this  section  of  the  RFP  early  in 
the  program.  The  Deputy  for  T&E  ensures  that  his 
office  and  all  associated  test  organizations  requir¬ 
ing  the  information  receive  the  test  documentation 
on  time.  Usually,  the  contractor  sends  the  data  pack¬ 
ages  directly  to  the  Deputy  for  T&E,  who,  in  turn, 
has  a  distribution  list  trimmed  to  the  minimum  num¬ 
ber  of  copies  for  agencies  needing  that  information 
to  perform  their  mission  and  oversight  responsibili¬ 
ties.  It  is  important  for  the  Deputy  for  T&E  to  use 
an  integrated  test  program  and  request  contractor 
test  plans  and  procedures  well  in  advance  of  the 
actual  test  performance  to  ensure  that  the  office  of 
the  Deputy  for  T&E  has  time  to  approve  the  pro¬ 
cedures  or  implement  modifications. 

Conversely,  the  Deputy  for  T&E  must  receive  the 
test  results  and  reports  on  time  to  enable  the  office 


of  the  Deputy  for  T&E,  the  PM,  and  higher  au¬ 
thorities  to  make  program  decisions.  Further,  the 
data  received  should  be  tailored  to  provide  the 
minimum  information  needed.  The  Deputy  for 
T&E  must  be  aware  that  data  requirements  in  ex¬ 
cess  of  the  minimum  needed  may  lead  to  an  un¬ 
necessary  increase  in  overall  program  cost.  For 
data  that  is  needed  quickly  and  informally  (at  least 
initially),  the  Deputy  for  T&E  can  request  Quick- 
Look  Reports  that  give  test  results  immediately 
after  test  performance.  The  Deputy  for  T&E  is 
also  responsible  for  coordinating  with  the  contrac¬ 
tor  on  all  report  formats  (the  in-house  contractor 
format  is  acceptable  in  most  cases). 

The  contract  must  specify  the  data  the  contractor 
will  supply  the  Operational  Test  Agency  (OTA). 
Unlike  Development  Test  and  Evaluation  (DT&E), 
the  contractor  will  not  be  making  the  Operational 
Test  and  Evaluation  (OT&E)  plans,  procedures  or 
reports.  These  documents  are  the  responsibility  of 
the  OTA.  The  PMO  Deputy  for  T&E  should  in¬ 
clude  the  OTA  on  the  distribution  list  for  all  test 
documents  that  are  of  concern  during  the  DT&E 
phase  of  testing  so  they  will  be  informed  of  test 
item  progress  and  previous  testing.  In  this  way, 
the  OTA  will  be  informed  when  developing  their 
own  test  plans  and  procedures  for  OT&E.  In  fact, 
OTA  representatives  should  attend  the  CDRL 
Review  Board  and  provide  the  PMO  with  a  list 
of  the  types  of  documents  the  OTA  will  need.  The 
Deputy  for  T&E  should  coordinate  the  test  sec¬ 
tions  of  this  data  list  with  the  OTA  and  indicate 
concerns  at  that  meeting.  All  contractor  test  re¬ 
ports  should  be  made  available  to  the  OTA.  In 
return,  the  Deputy  for  T&E  must  stay  informed 
of  all  OTA  activities,  understand  their  test  proce¬ 
dures,  and  plan  and  receive  their  test  reports.  Un¬ 
like  DT&E,  the  PMO  Deputy  for  T&E  will  not 
have  report  or  document  approval  authority  for 
OT&E  items  as  he/she  does  over  contractor  docu¬ 
mentation.  The  Deputy  for  T&E  is  always  respon¬ 
sible  for  keeping  the  PM  informed  of  OT&E 
results. 
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4.3.3  Test  Schedule  Formulation 

A  very  important  task  the  Deputy  for  T&E  has 
during  the  creation  of  the  RFP  is  the  test  program 
schedule.  Initially,  the  PM  will  need  contractor  pre¬ 
dictions  of  the  hardware  (and  software  in  some 
cases)  availability  dates  for  models,  prototypes, 
mock-ups,  full-scale  models,  etc.,  once  the  con¬ 
tract  is  awarded.  The  Deputy  for  T&E  uses  this 
information  and  the  early  test  planning  in  the  Tech¬ 
nology  Development  Strategy  (TDS)  document 
to  create  a  realistic  front-end  schedule  of  the  in- 
house  testing  the  contractor  will  conduct  before 
government  testing  (Development  Testing  (DT) 
and  Operational  Testing  (OT)).  Then,  a  draft  inte¬ 
grated  schedule  (Part  II,  TEMP)  is  developed  upon 
which  the  government  DT  and  OT  schedules  can 
be  formulated  and  contractor  support  requirements 
determined.  The  Deputy  for  T&E  can  use  past 
experience  in  testing  similar  weapon  systems/ 
acquisition  items  or  contract  test  organizations  that 
have  the  required  experience  to  complete  the  en¬ 
tire  test  schedule.  Since  the  test  schedule  is  a  criti¬ 
cal  contractual  item,  contractor  input  is  very  im¬ 
portant.  The  test  schedule  will  normally  become 
an  item  for  negotiation  once  the  RFP  is  released, 
and  the  contractor’s  proposal  is  received.  Atten¬ 
tion  must  be  given  to  ensuring  the  test  schedule  is 
not  so  success-oriented  that  retesting  of  failures 
cause  serious  program  delays  for  either  the  gov¬ 
ernment  test  agencies  or  the  contractor. 

Another  important  early  activity  the  Deputy  for 
T&E  must  accomplish  is  to  coordinate  the  OT&E 
test  schedule.  Since  the  contractor  may  be  required 
to  provide  support,  the  OT&E  test  support  may 
need  to  be  contractually  agreed  upon  before  con¬ 
tract  award.  Sometimes,  the  Deputy  for  T&E  can 
formulate  a  straw  man  schedule  (based  on  previ¬ 
ous  experience)  and  present  this  schedule  to  the 
operational  test  representative  at  the  initial  T&E 
Integrated  Product  Team  (IPT)  meeting  for  review; 
or  the  Deputy  for  T&E  can  contact  the  OTA  and 
arrange  a  meeting  to  discuss  the  new  program.  In 
the  meeting,  time  requirements  envisioned  by  OTA 
can  be  discussed.  Input  from  that  meeting  then 


goes  into  the  RFP  and  to  the  PM.  The  test  sched¬ 
ule  must  allow  time  for  DT&E  testing  and  OT&E 
testing  when  testing  is  not  combined  or  test  assets 
are  not  limited.  Before  setup  of  Initial  Operational 
Test  and  Evaluation  (IOT&E),  certification  of 
readiness  for  IOT&E  may  require  a  time  gap  for 
review  of  DT&E  test  results  and  refurbishment  or 
corrections  of  deficiencies  discovered  during 
DT&E,  etc.  The  test  schedule  for  DT&E  should 
not  be  over-run  so  that  the  IOT&E  test  schedule 
is  adversely  impacted,  reducing  program  sched¬ 
ule  time  with  inadequate  operational  testing  or 
rushing  the  reporting  of  IOT&E  results.  For  ex¬ 
ample,  if  the  DT&E  schedule  slips  six  months, 
the  OT&E  schedule  and  milestone  decision  should 
slip  also.  The  IOT&E  should  not  be  shortened  just 
to  make  a  milestone  decision  date. 

4.3.4  Programmatic  Environmental  Analysis 

The  PMO  personnel  should  be  sensitive  to  the 
potential  environmental  consequences  of  system 
materials,  operations  and  disposal  requirements. 
Public  Laws  (Title  40,  Code  of  Federal  Regula¬ 
tions  (CFR),  Parts  1500-1508;  National  Environ¬ 
mental  Policy  Act  (NEPA)  Regulations;  Execu¬ 
tive  Order  12114,  Environmental  Effects  Abroad 
of  Major  Federal  Actions,  4  Jan  1979;  DoD  5000 
Series;  etc.)  require  analysis  of  hazardous  materi¬ 
als  and  appropriate  mitigation  measures  during 
each  acquisition  phase.  As  stated  in  DoD  5000.2- 
R,  “Planning  shall  consider  the  potential  testing 
impacts  on  the  environment.” 

Litigation  resulting  in  personal  fines  and  impris¬ 
onment  successfully  executed  against  government 
employees  have  raised  the  environmental  aware¬ 
ness  at  test  ranges  and  facilities.  Environmental 
Impact  Statements  (EISs)  (supported  by  long,  thor¬ 
ough  studies  and  public  testimony)  or  Environ¬ 
mental  Analysis  and  Assessments  are  generally  re¬ 
quired  before  any  system  testing  can  be  initiated. 
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4.4  PMO/CONTRACTOR  TEST 
MANAGEMENT 

The  PMO  will,  in  most  cases,  have  a  contractor 
test  section  counterpart.  With  this  counterpart,  the 
Deputy  for  T&E  works  out  the  detailed  test  plan¬ 
ning,  creation  of  schedules,  etc.,  for  the  entire  test 
program.  The  PMO  uses  input  from  all  sources 
(contracts,  development  test  agencies,  OTAs,  higher 
headquarters,  etc.)  to  formulate  the  test  program’s 
length,  scope,  and  necessary  details.  The  Deputy 
for  T&E  ensures  that  the  RFP  reflects  the  test  pro¬ 
gram  envisioned  and  the  contractor’s  role  in  the 
acquisition  process.  The  Deputy  for  T&E  also  en¬ 
sures  the  RFP  includes  provisions  for  government 
attendance  at  contractor’s  tests  and  that  all  contrac¬ 
tor  test  results  are  provided  to  the  government. 

After  the  RFP  has  been  issued  and  the  contractor 
has  responded,  the  proposal  is  reviewed  by  the 
PMO.  The  Deputy  for  T&E  is  responsible  for 
performing  a  technical  evaluation  on  the  test  por¬ 
tions  of  the  proposal.  In  this  technical  evaluation, 
the  Deputy  for  T&E  compares  the  proposal  to  the 
SOW,  test  schedule,  IFPP,  etc.,  and  reviews  the 
contractor’s  cost  of  each  testing  item.  This  is  an 
iterative  process  of  refining,  clarifying  and  modi¬ 
fying  that  will  ensure  the  final  contract  between 
the  PMO  and  the  prime  contractor  (subcontrac¬ 
tors)  contains  all  test-related  tasks  and  is  priced 
within  scope  of  the  proposed  test  program.  Once 
technical  agreement  on  the  contractor’s  technical 
approach  is  reached,  the  Deputy  for  T&E  is  re¬ 
sponsible  for  giving  inputs  to  the  government  con¬ 
tracting  officer  during  contract  negotiations.  The 
contracting  officer-requested  contract  deliverables 
are  assigned  Contract  Line  Item  Numbers 
(CLINs),  which  are  created  by  the  Deputy  for 
T&E.  This  will  ensure  the  contractor  delivers  the 
required  performances  at  specified  intervals  dur¬ 
ing  the  life  of  the  contract.  Usually,  there  will  be 
separate  contracts  for  development  and  produc¬ 
tion  of  the  acquisition  item.  For  each  type  of  con¬ 
tract,  the  Deputy  for  T&E  has  the  responsibility 
to  provide  the  PCO  and  PM  with  the  test  and 
evaluation  input. 


4.5  INTEGRATED  PRODUCT  TEAMS 
FOR  TEST  AND  EVALUATION 

Before  the  final  version  of  the  RFP  is  created,  the 
Deputy  for  T&E  should  form  an  IPT,  a  test 
planning/integration  working  group.  This  group 
includes  the  OTA,  development  test  agency,  or¬ 
ganizations  that  may  be  jointly  acquiring  the  same 
system,  the  test  supporting  agencies,  operational 
users,  and  any  other  organizations  that  will  be  in¬ 
volved  in  the  test  program  by  providing  test  sup¬ 
port  or  by  conducting,  evaluating,  or  reporting  on 
testing.  The  functions  of  the  groups  are  to:  facili¬ 
tate  the  use  of  testing  expertise,  instrumentation, 
facilities,  simulations  and  models;  integrate  test 
requirements;  accelerate  the  TEMP  coordination 
process;  resolve  test  cost  and  scheduling  problems; 
and  provide  a  forum  to  ensure  T&E  of  the  system 
is  coordinated.  The  existence  of  a  test  coordinat¬ 
ing  group  does  not  alter  the  responsibilities  of  any 
command  or  headquarters;  and  in  the  event  of  dis¬ 
agreement  within  a  group,  the  issue  is  resolved 
through  the  normal  command/staff  channels.  In 
later  meetings,  the  contractor  participates  in  this 
test  planning  group;  however,  the  contractor  may 
not  be  selected  by  the  time  the  first  meetings  are 
held. 

The  purposes  of  these  meetings  are  to  review  and 
assist  in  the  development  of  early  test  documenta¬ 
tion,  the  TEMP,  and  to  agree  on  basic  test  pro¬ 
gram  schedules,  scope,  support,  etc.  The  TEMP 
serves  as  the  top-level  test  management  document 
for  the  acquisition  program,  being  updated  as  the 
changing  program  dictates. 

4.6  TEST  PROGRAM  FUNDING/ 
BUDGETING 

The  PMO  must  identify  funds  for  testing  very  early 
so  that  test  resources  can  be  obtained.  The  Deputy 
for  T&E  uses  the  acquisition  schedule,  TEMP,  and 
other  program  and  test  documentation  to  identify 
test  resource  requirements.  The  Deputy  for  T&E 
coordinates  these  requirements  with  the  contractor 
and  government  organizations  that  have  the  test 
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facilities  to  ensure  their  availability  for  testing.  The 
Deputy  for  T&E  ensures  that  test  costs  include  con¬ 
tractor  and  government  test  costs.  The  contractor’s 
test  costs  are  normally  outlined  adequately  in  his 
proposal;  however,  the  government  test  ranges,  in¬ 
strumentation  and  test-support  resource  costs  must 
be  determined  by  other  means.  Usually,  the  Deputy 
for  T&E  contacts  the  test  organization  and  outlines 
the  test  program  requirements;  and  the  test  organi¬ 
zation  sends  the  program  office  an  estimate  of  the 
test  program  costs.  The  Deputy  for  T&E  then  ob¬ 
tains  cost  estimates  from  all  test  sources  that  the 
Deputy  for  T&E  anticipates  using  and  supplies  this 
information  to  the  PM.  The  Deputy  for  T&E  must 
also  ensure  that  any  program  funding  reductions 
are  not  absorbed  entirely  by  the  test  program.  Some 
cutbacks  may  be  necessary  and  allowable;  but  the 
test  program  must  supply  the  PM,  other  defense 
decision-making  authorities,  and  the  Congress  with 
enough  information  to  make  program  milestone 
decisions. 

4.7  TECHNICAL  REVIEWS,  DESIGN 
REVIEWS,  AND  AUDITS 

The  role  of  the  Deputy  for  T&E  changes  slightly 
during  the  contractor’s  technical  reviews,  design 
reviews,  physical  and  functional  configuration  au¬ 
dits,  etc.  Usually,  the  Deputy  for  T&E  plans,  di¬ 
rects  or  monitors  government  testing;  however,  in 
the  reviews  and  audits,  the  Deputy  examines  the 
contractor’s  approach  to  the  test  problem  and 
evaluates  the  validity  of  the  process  and  the  accu¬ 
racy  of  the  contractor’s  results.  The  Deputy  for 
T&E  uses  personal  experience  and  background 
in  T&E  to  assess  whether  the  contractor  did 
enough  or  too  little  testing;  whether  the  tests  were 
biased  in  any  way;  and  if  they  followed  a  logical 
progression  using  the  minimum  of  time,  effort  and 
funds.  If  the  Deputy  for  T&E  finds  any  discrep¬ 
ancies,  the  Deputy  must  inform  the  contractor,  the 
PM  and  the  PCO  to  validate  the  conclusions  be¬ 
fore  effecting  corrections.  Each  type  of  review  or 
audit  will  have  a  different  focus/orientation,  but 
the  Deputy  for  T&E  will  always  be  concerned 
with  the  testing  process  and  how  it  is  carried  out. 


After  each  review,  the  Deputy  for  T&E  should 
always  document  all  observations  for  future 
reference. 

4.8  CONTRACTOR  TESTING 

The  Deputy  for  T&E  is  responsible  for  ensuring 
that  contractor-conducted  tests  are  monitored  by 
the  government.  The  Deputy  for  T&E  must  also 
be  given  access  to  all  contractor  internal  data,  test 
results  and  test  reports  related  to  the  acquisition 
program.  Usually,  the  contract  requires  that  gov¬ 
ernment  representatives  be  informed  ahead  of  time 
of  any  (significant  or  otherwise)  testing  the  con¬ 
tractor  conducts  so  the  government  can  arrange 
to  witness  certain  testing  or  receive  results  of  the 
tests.  Further,  the  contractor’s  internal  data  should 
be  available  as  a  contract  provision.  The  Deputy 
for  T&E  must  ensure  that  government  test  per¬ 
sonnel  (DT&E/OT&E)  have  access  to  contractor 
test  results.  It  would  be  desirable  to  have  all  testers 
observe  some  contractor  tests  to  help  develop  con¬ 
fidence  in  the  results  and  identify  areas  of  risk. 

4.9  SPECIFICATIONS 

Within  the  program  office,  the  engineering  sec¬ 
tion  is  usually  tasked  to  create  the  system  perfor¬ 
mance  specifications  for  release  of  the  RFP.  The 
contractor  is  then  tasked  with  creating  the  specifi¬ 
cation  documentation  called  out  by  the  contract, 
which  will  be  delivered  once  the  item/system  de¬ 
sign  is  formalized  for  production.  The  Deputy  for 
T&E  performs  an  important  function  in  specifica¬ 
tion  formulation  by  reviewing  the  specifications 
to  determine  if  performance  parameters  are  test¬ 
able;  if  current,  state-of-the-art  technology  can 
determine  (during  the  DT&E  test  phase)  if  the 
performance  specifications  are  being  met  by  the 
acquisition  item;  or  if  the  specified  parameters  are 
too  “tight.”  A  specification  is  too  “tight”  if  the 
requirements  (Section  3)  are  impossible  to  meet 
or  demonstrate,  if  the  specification  has  no  impact 
on  the  form,  fit,  or  function  of  the  end-item,  the 
system  it  will  become  a  part  of  or  the  system  with 
which  it  will  interact.  The  Deputy  for  T&E  must 
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determine  if  test  objectives  can  be  adequately  for¬ 
mulated  from  those  specifications  that  will  pro¬ 
vide  thresholds  of  performance,  minimum  and 
maximum  standards,  and  reasonable  operating  con¬ 
ditions  for  the  end-item’s  final  mission  and  oper¬ 
ating  environment.  The  specifications  (Section  4) 
shape  the  DT&E  testing  scenario,  test  ranges,  test 
support,  targets,  etc.,  and  are  very  important  to 
the  Deputy  for  T&E. 

4.10  INDEPENDENT  TEST  AND 
EVALUATION  AGENCIES 

The  PMO  Deputy  for  T&E  does  not  have  direct 
control  over  government-owned  test  resources, 
test  facilities,  test  ranges,  test  personnel,  etc. 
Therefore,  the  Deputy  for  T&E  must  depend  on 
those  DT  or  OT  organizations  controlling  them 
and  stay  involved  with  the  test  agency  activities. 
The  amount  of  involvement  depends  on  the  item 
being  tested;  its  complexity,  cost,  and  character¬ 
istics;  the  length  of  time  for  testing;  amount  of 
test  funds;  etc.  Usually,  the  “nuts  and  bolts”  de¬ 
tailed  test  plans  and  procedures  are  written  by 
the  test  organizations  controlling  the  test  resources 
with  input  and  guidance  from  the  Program  Of¬ 
fice  Deputy  for  T&E.  The  Deputy  for  T&E  is 


responsible  for  ensuring  that  the  tests  (DT&E)  are 
performed  using  test  objectives  based  on  the  speci¬ 
fications  and  that  the  requirements  of  timeliness, 
accuracy,  and  minimal  costs  are  met  by  the  test 
program  design.  During  the  testing,  the  Deputy 
for  T&E  monitors  test  results.  The  test  agencies 
(DT/OT)  submit  a  copy  of  their  report  to  the  Pro¬ 
gram  Office  at  the  end  of  testing,  usually  to  the 
Office  of  the  Deputy  for  T&E.  The  Army  is  the 
only  Service  to  have  a  designated  independent 
evaluation  agency  which  provides  feedback  to  the 
program  office. 

4.11  PMO  RESPONSIBILITIES  FOR 
OPERATIONAL  TEST  AND 
EVALUATION  (OT&E) 

In  the  government  PMO,  there  should  be  a  sec¬ 
tion  responsible  for  T&E.  Besides  being  respon¬ 
sible  for  DT&E  support  to  the  PM,  this  section 
should  be  responsible  for  program  coordination 
with  the  OT&E  agency  (Figure  4-1).  The  offices 
of  the  systems  engineer  or  the  Deputy  for  T&E 
may  be  designated  to  provide  this  support  to  the 
PM.  In  some  Services,  responsibilities  of  the 
Deputy  for  T&E  include  coordination  of  test  re¬ 
sources  for  all  phases  of  OT&E. 


•  UNDERSTAND  THE  POLICIES 

•  ORGANIZE  FOR  T&E 

•  KEEP  SYSTEM  REQUIREMENTS  DOCUMENTS  CURRENT 

•  AGONIZE  OVER  SYSTEMTHRESHOLDS 

•  WORK  CLOSELY  WITH  THE  OPERATIONAL  TEST  DIRECTOR 

•  DON’T  FORGET  ABOUT  OPERATIONAL  SUITABILITY 

•  MAKE  FINAL  DT&E  A  REHEARSAL  FOR  IOT&E 

•  PREPARE  INTERFACING  SYSTEMS  F0RY0UR  IOT&E 

•  MANAGE  SOFTWARETESTING  CLOSELY 

•  TRACK  AVAILABILITY  OFTEST  RESOURCES  ANDTEST  SUPPORT 
PERSONNEL/FACILITIES 

SOURCE:  Matt  Reynolds,  NAVSEA. 

Figure  4-1.  Lessons  Learned  from  OT&E  for  the  PM 
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4.11.1  Contract  Responsibilities 

The  Deputy  for  T&E  or  a  T&E  representative 
ensures  that  certain  sections  of  the  RFP  contain 
sufficient  allowance  for  T&E  support  by  contrac¬ 
tors.  This  applies  whether  the  contract  is  for  a 
development  item,  a  production  item  (limited  pro¬ 
duction,  such  as  Low  Rate  Initial  Production 
(LRIP)  or  Full  Rate  Production  (FRP))  or  the  en¬ 
hancement/upgrade  of  portions  of  a  weapons  sys¬ 
tem.  Where  allowed  within  the  law,  contractor 
support  for  OT&E  should  be  considered  to  help 
resolve  basic  issues  such  as  data  collection  require¬ 
ments,  test  resources,  contractor  test  support,  and 
funding. 

In  the  overall  portion  of  the  RFP,  government 
personnel,  especially  those  in  the  OTAs,  must  be 
guaranteed  access  to  the  contractor’s  development 
facilities,  particularly  during  the  DT&E  Phase. 
Government  representatives  must  be  allowed  to 
observe  all  contractor  in-house  testing  and  have 
access  to  test  data  and  reports. 

4.11.2  Data  Requirements 

The  contract  must  specify  the  data  the  contractor 
will  supply  the  OTA.  Unlike  DT&E,  the  contrac¬ 
tor  will  not  be  making  the  OT&E  plans,  proce¬ 
dures  or  reports.  These  documents  are  the  respon¬ 
sibility  of  the  OTA.  The  PMO  Deputy  for  T&E 
should  include  the  OTA  on  the  distribution  list 
for  all  test  documents  that  are  of  concern  during 
the  DT&E  phase  of  testing  so  they  will  be  in¬ 
formed  of  test  item  progress  and  previous  testing. 
In  this  way,  the  OTA  will  be  informed  when  de¬ 
veloping  their  own  test  plans  and  procedures 
for  OT&E.  In  fact,  OTA  representatives  should 
attend  the  CDRL  Review  Board  and  provide  the 
PMO  with  a  list  of  the  types  of  documents  the 
OTA  will  need.  The  Deputy  for  T&E  should  co¬ 
ordinate  the  test  sections  of  this  data  list  with  the 
OTA  and  indicate  concerns  at  that  meeting.  All 
contractor  test  reports  should  be  made  available 
to  the  OTA.  In  return,  the  Deputy  for  T&E  must 
stay  informed  of  all  OTA  activities,  understands 


their  test  procedures  and  plans  and  receives  their 
test  reports.  Unlike  DT&E,  the  PMO  Deputy  for 
T&E  will  not  have  report  or  document  approval 
authority  as  the  Deputy  for  T&E  does  over  con¬ 
tractor  documentation.  The  Deputy  for  T&E  is 
always  responsible  for  keeping  the  PM  informed 
of  OT&E  results. 

4.11.3  Test  Schedule 

Another  important  early  activity  the  Deputy  for 
T&E  must  accomplish  is  to  coordinate  the  OT&E 
test  schedule.  Since  the  contractor  may  be  required 
to  provide  support,  the  OT&E  test  support  may 
need  to  be  contractually  agreed  upon  before  con¬ 
tract  award.  Sometimes,  the  Deputy  for  T&E  can 
formulate  a  draft  schedule  (based  on  previous  ex¬ 
perience)  and  present  this  schedule  to  the  opera¬ 
tional  test  representative  at  the  initial  Test  Plan¬ 
ning  Working  Group  (TPWG)  for  review;  or  the 
Deputy  for  T&E  can  contact  the  OTA  and  arrange 
a  meeting  to  discuss  the  new  program.  In  the 
meeting,  time  requirements  envisioned  by  OTA 
can  be  discussed.  Input  from  that  meeting  then 
goes  into  the  RFP  and  to  the  PM.  The  test  sched¬ 
ule  must  allow  time  for  DT&E  testing  and  OT&E 
testing  if  testing  is  not  combined  or  test  assets  are 
limited.  Before  setup  of  IOT&E,  the  Certification 
of  Readiness  for  IOT&E  process  may  require  a 
time  gap  for  review  of  DT&E  test  results  and  re¬ 
furbishment  or  corrections  of  deficiencies  discov¬ 
ered  during  DT&E,  etc.  The  test  schedule  for 
DT&E  should  not  be  so  “success-oriented”  that 
the  IOT&E  test  schedule  is  adversely  impacted, 
not  allowing  enough  time  for  adequate  operational 
testing  or  the  reporting  of  IOT&E  results. 

4.11.4  Contractor  Support 

The  Deputy  for  T&E  provides  all  T&E  input  to 
the  RFP/SOW.  The  Deputy  for  T&E  must  deter¬ 
mine,  before  the  beginning  of  the  program  acqui¬ 
sition  phase,  whether  the  contractor  will  be  in¬ 
volved  in  supporting  IOT&E  and,  if  so,  to  what 
extent.  According  to  Title  10,  U.S.C.,  the  system 
contractor  can  only  be  involved  in  the  conduct  of 
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IOT&E  if,  once  the  item  is  fielded,  tactics  and 
doctrine  say  the  contractor  will  be  providing  sup¬ 
port  or  operating  that  item  during  combat.  If  not, 
no  system  contractor  support  is  allowed  during 
IOT&E.  Before  IOT&E;  however,  the  contractor 
may  be  tasked  with  providing  training,  training 
aids,  and  handbooks  to  Service  training  cadre  so 
they  can  train  the  IOT&E  users  and  maintenance 
personnel.  In  addition,  the  contractor  must  be  re¬ 
quired  to  provide  sufficient  spare  parts  for  the  op¬ 
erational  maintenance  personnel  to  maintain  the 
test  item  while  undergoing  operational  testing. 
These  support  items  must  be  agreed  upon  by  the 
PMO  and  OTA  and  must  contractually  bind  the 
contractor.  If,  however,  the  contractor  will  be  re¬ 
quired  to  provide  higher-level  maintenance  of  the 
item  for  the  duration  of  the  IOT&E,  data  collec¬ 
tion  on  those  functions  will  be  delayed  until  a 
subsequent  Follow-on  Operational  Test  and 
Evaluation  (FOT&E). 

4.11.5  Statement  of  Work  (SOW) 

One  of  the  most  important  documents  receiving 
input  from  the  Deputy  for  T&E  is  the  SOW.  The 
Deputy  for  T&E  must  outline  all  required  or  an¬ 
ticipated  contractor  support  for  DT&E  and  OT&E. 
This  document  outlines  data  requirements,  con¬ 
tractor-conducted  or  supported  testing,  govern¬ 
ment  involvement  (access  to  contractor  data,  tests, 
and  results),  operational  test  support,  and  any  other 
specific  test  requirements  the  contractor  will  be 
tasked  to  perform  during  the  duration  of  the 
contract. 

4.11.6  Operational  OT&E  Funding 

The  Deputy  for  T&E  provides  the  PM  estimates 
of  PMO  test  program  costs  to  conduct  OT&E. 
This  funding  includes  contractor  and  government 
test  support  for  which  the  program  office  directly 
or  indirectly  will  be  responsible.  Since  Service 
OTAs  fund  differently,  program  office  funding  for 
conducting  OT&E  varies.  The  Deputy  for  T&E 
must  determine  these  costs  and  inform  the  PM. 


4. 1 1 .7  Test  and  Evaluation  Master  Plan 
(TEMP) 

The  Deputy  for  T&E  is  responsible  for  managing 
the  TEMP  throughout  the  test  program.  The  OTA 
usually  is  tasked  to  complete  the  OT  section  of 
the  TEMP  and  outline  their  proposed  test  program 
through  all  phases  of  OT&E.  The  resources  re¬ 
quired  for  OT&E  should  also  be  entered  in  Part 
V.  It  is  important  to  keep  the  TEMP  updated  regu¬ 
larly  so  that  test  organizations  involved  in  OT&E 
understand  the  scope  of  their  test  support.  The 
TEMP  should  be  updated  regularly  by  the  OTA. 
Further,  if  any  upgrades,  improvements  or  en¬ 
hancements  to  the  fielded  weapon  system  occur, 
the  TEMP  must  be  updated  or  a  new  one  created 
to  outline  new  DT  and  OT  requirements. 

4.11.8  Program  Management  Office 
Support  for  OT&E 

Even  though  operational  testing  is  performed  by 
an  independent  organization,  the  PM  plays  an  im¬ 
portant  role  in  its  planning,  reporting  and  fund¬ 
ing.  The  PM  must  coordinate  program  activities 
with  the  test  community  in  the  TE  Working-level 
Integrated  Product  Team  (WIPT),  especially  the 
OTAs.  The  PM  ensures  that  testing  can  address 
the  critical  issues,  and  provides  feedback  from 
OT&E  testing  activities  to  contractors. 

At  each  milestone  review,  the  PM  is  required  to 
brief  the  decision  authority  on  the  testing  planned 
and  completed  on  the  program.  It  is,  therefore, 
important  that  PMO  personnel  have  a  good  un¬ 
derstanding  of  the  test  program  and  that  they  work 
with  the  operational  test  community.  This  will 
ensure  OT&E  is  well-planned  and  adequate  re¬ 
sources  are  available.  The  PMO  should  involve 
the  test  community  by  organizing  test  coordinat¬ 
ing  groups  at  program  initiation  and  by  establish¬ 
ing  channels  of  communication  between  the  PMO 
and  the  key  test  organizations.  The  PMO  can  of¬ 
ten  avoid  misunderstandings  by  aggressively  moni¬ 
toring  the  system  testing  and  providing  up-to-date 
information  to  key  personnel  in  the  Office  of  the 
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Secretary  of  Defense  and  the  Services.  The  PMO 
staff  should  keep  appropriate  members  of  the  test 
community  well-informed  concerning  system 
problems  and  the  actions  taken  by  the  PMO  to 
correct  them.  The  PMO  must  assure  that  contrac¬ 
tor  and  government  DT&E  supports  the  decision 
to  certify  the  system’s  readiness  for  IOT&E. 

4.11.9  Support  for  IOT&E 

The  Certification  of  Readiness  for  IOT&E  pro¬ 
cess  provides  a  forum  for  a  final  review  of  test 
results  and  support  required  by  the  for  IOT&E. 
The  Deputy  for  T&E  must  ensure  the  contract 
portions  adequately  cover  the  scope  of  testing  as 
outlined  by  the  OTA.  The  program  office  may 
want  to  provide  an  observer  to  represent  the 
Deputy  for  T&E  during  the  actual  testing.  The 
Deputy  for  T&E  involvement  in  IOT&E  will  be 
to  monitor  and  coordinate;  the  Deputy  for  T&E 
will  keep  the  PM  informed  of  progress  and  prob¬ 
lems  that  arise  during  testing  and  will  monitor  re¬ 
quired  PMO  support  to  the  test  organization.  Also, 
enough  LRIP  items  must  be  manufactured  to  run 
a  complete  and  adequate  OT&E  program.  The 
DOT&E  determines  the  number  of  LRIP  items 
for  IOT&E  on  oversight  programs  and  the  OTA 
makes  this  determination  for  all  others.  (Reference 
16)  For  problems  requiring  program  office  action, 
the  Deputy  for  T&E  will  be  the  point  of  contact. 

The  Deputy  for  T&E  will  be  concerned  with 
IOT&E  of  the  LRIP  units  after  a  limited  number 
are  produced.  The  IOT&E  must  be  closely  moni¬ 
tored  so  that  a  full-rate  production  decision  can 
be  made.  As  in  the  operational  assessments,  the 
Deputy  for  T&E  will  be  monitoring  test  proce¬ 
dures  and  results  and  keeping  the  PM  informed. 
If  the  item  does  not  succeed  during  IOT&E,  a  new 
process  of  DT&E  or  a  modification  may  result; 
and  the  Deputy  for  T&E  will  be  involved  (as  in 
any  new  programs  inception).  If  the  item  passes 
IOT&E  testing  and  is  produced  at  full  rate,  the 
Deputy  for  T&E  will  be  responsible  for  ensuring 
that  testing  of  those  production  items  is  adequate 


to  ensure  that  the  end  items  physically  and  func¬ 
tionally  resemble  the  development  items. 

4.11.10  FOT&E  and  Modifications, 
Upgrades,  Enhancements,  or 
Additions 

During  FOT&E,  the  Deputy  for  T&E  monitors  the 
testing  and  the  level  of  contractor  involvement  is 
usually  negotiated.  The  Deputy  for  T&E  should 
receive  any  reports  generated  by  the  operational 
testers  during  this  time.  Any  deficiencies  noted  dur¬ 
ing  FOT&E  should  be  evaluated  by  the  PMO, 
which  may  decide  to  incorporate  upgrades,  en¬ 
hancements  or  additions  to  the  current  system  or  in 
future  increments.  If  the  PM  and  the  engineering 
section  of  the  program  office  design  or  develop 
modifications  that  are  incorporated  into  the  weapon 
system  design,  additional  FOT&E  may  be  required. 

Once  a  weapon  system  is  fielded,  portions  of  that 
system  may  become  obsolete,  ineffective,  or  defi¬ 
cient  and  may  need  replacing,  upgrading  or  en¬ 
hancing  to  ensure  the  weapon  system  meets  cur¬ 
rent  and  future  requirements.  The  Deputy  for  T&E 
plays  a  vital  role  in  this  process.  Modifications  to 
existing  weapon  systems  may  be  managed  as  an 
entire  newly  acquired  weapon  system.  However, 
since  these  are  changes  to  existing  systems,  the 
Deputy  for  T&E  is  responsible  for  determining  if 
these  enhancements  degrade  the  existing  system, 
are  compatible  with  its  interfaces  and  functions 
and  whether  Non-developmental  Items  (NDIs) 
require  retest  or  the  entire  weapon  system  needs 
re- verification.  The  Deputy  for  T&E  must  plan 
the  test  program’s  funding,  schedule,  test  program 
and  contract  provisions  with  these  items  in  mind. 
A  new  TEMP  may  have  to  be  generated  or  the 
original  weapon  system  TEMP  modified  and  re¬ 
coordinated  with  the  test  organizations.  The  de¬ 
sign  of  the  DT&E  and  FOT&E  program  usually 
requires  coordination  with  the  engineering,  con¬ 
tracting,  and  program  management  IPTs  of  the 
program  office. 
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4.11.11  Test  Resources 

During  all  phases  of  OT,  the  Deputy  for  T&E  must 
coordinate  with  the  operational  testers  to  ensure 
they  have  the  test  articles  needed  to  accomplish 
their  mission.  Test  resources  will  be  either  con¬ 
tractor  provided  or  government  provided.  The 
contractor  resources  must  be  covered  in  the  con¬ 
tract,  whether  in  the  development  contract  or  the 
production  contract.  Government  test  resources 
needed  are  determined  by  the  operational  testers. 
They  usually  coordinate  the  test  ranges,  test  sup¬ 
port,  and  the  user  personnel  for  testing.  The  PM 
programs  funding  for  his  support  of  OT.  Funding 
for  OT&E  is  identified  in  the  TEMP  and  funded 
in  the  PMO’s  budget.  The  Services’  policy  specify 
how  the  OTAs  develop  and  manage  their  own 
budgets  for  operational  testing. 

4.12  SUMMARY 

Staffing  requirements  in  the  PMO  vary  with  the 
program  phase  and  the  T&E  workload.  T&E 
expertise  is  essential  in  the  early  planning  stages 


but  can  be  provided  through  matrix  support.  The 
Deputy  for  T&E  may  be  subordinate  to  the  chief 
engineer  in  early  phases  but  should  become  a  sepa¬ 
rate  staff  element  after  prototype  testing.  Changing 
of  critical  players  can  destroy  established  working 
relationships  and  abrogate  prior  agreements  if  con¬ 
tinuity  is  not  maintained.  The  PMO  management 
of  T&E  must  provide  for  an  integrated  focus  and  a 
smooth  transition  from  one  staff-support  mode  to 
the  next. 

The  PMO  should  be  proactive  in  its  relations  with 
the  Service  OTA.  There  are  many  opportunities 
to  educate  the  OTA  on  system  characteristics  and 
expected  performance.  Early  OTA  input  to  design 
considerations  and  requirements  clarification  can 
reduce  downstream  surprises.  Operational  testing 
is  an  essential  component  of  the  system  develop¬ 
ment  and  decision-making  process.  It  can  be  used 
to  facilitate  system  development  or  may  become 
an  impediment.  In  many  cases,  the  PMO  attitude 
toward  operational  testing  and  the  OTA  will  in¬ 
fluence  which  role  the  OTA  assumes. 
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TEST-RELATED 

DOCUMENTATION 


5.1  INTRODUCTION 

During  the  course  of  a  defense  acquisition  pro¬ 
gram,  many  documents  are  developed  that  have 
significance  for  those  responsible  for  testing  and 
evaluating  the  system.  This  chapter  is  designed  to 
provide  background  on  some  of  these  documents. 

As  Figure  5-1  shows,  test-related  documentation 
spans  a  broad  range  of  materials.  It  includes  re¬ 
quirements  documentation  such  as  the  Capability 
Development  Document  (CDD)  and  Capability 
Production  Document  (CPD);  program  decision 
documentation  such  as  Acquisition  Decision  Memo¬ 
randum  (ADM)  with  exit  criteria;  and  program 
management  documentation  such  as  the  Acquisi¬ 
tion  Strategy,  Baseline  documentation,  the  Systems 
Engineering  Plan  (SEP),  the  logistics  support  plan¬ 
ning,  and  the  Test  and  Evaluation  Master  Plan 
(TEMP).  Of  importance  to  Program  Managers 
(PM)  and  to  Test  and  Evaluation  (T&E)  managers 
are  additional  test  program  documents  such  as  spe¬ 
cific  test  designs,  test  plans,  Outline  Test  Plans/Test 
Program  Outlines  (OTPs/TPOs),  evaluation  plans, 
and  test  reports.  This  chapter  concludes  with  a  de¬ 
scription  of  the  End-of-Test  Phase  and  Beyond  Low 
Rate  Initial  Production  (BLRIP)  Reports,  and  two 
special-purpose  T&E  status  reports  that  are  used  to 
support  the  milestone  decision  process. 

5.2  REQUIREMENTS 
DOCUMENTATION 

5.2.1  Continuing  Mission  Capability 
Analyses 

As  indicated  in  the  Chairman  of  the  Joint  Chiefs 
of  Staff  Instruction  (CJCSI)  3 170.0 ID,  Joint 


Capabilities  Integration  and  Development  System 
(JCIDS),  12  March  2004,  and  the  CJCS  Manual 
(CJCSM)  3 170.01  A  Operation  of  the  Joint  Ca¬ 
pabilities  Integration  and  Development  System,  12 
March  2004,  the  Joint  Chiefs,  via  the  Joint  Re¬ 
quirements  Oversight  Council  (JROC),  are  re¬ 
quired  to  conduct  continuing  mission  analyses  of 
their  assigned  areas  of  responsibility.  These  Func¬ 
tional  Analyses  (Area,  Needs,  Solutions)  may  re¬ 
sult  in  recommendations  to  initiate  new  acquisi¬ 
tion  programs  to  reduce  or  eliminate  operational 
deficiencies.  If  a  need  cannot  be  met  through 
changes  in  Doctrine,  Organization,  Training, 
Leadership  and  Education,  Personnel  and  Facili¬ 
ties  (DOTLPF)  and  a  Materiel  (M)  solution  is  re¬ 
quired,  the  needed  capability  is  described  first  in 
an  Initial  Capabilities  Document  (ICD)  and  then 
in  the  CDD.  When  the  cost  of  a  proposed  acqui¬ 
sition  program  is  estimated  to  exceed  limits  speci¬ 
fied  in  Department  of  Defense  Instruction  (DoDI) 
5000.2,  Operation  of  the  Defense  Acquisition  Sys¬ 
tem,  12  May  2003,  is  considered  a  Major  Defense 
Acquisition  Program  (MDAP)  and  requires  JROC 
approval  (JROC  Interest).  Lesser  programs  for  Ser¬ 
vice  action  are  designated  Joint  Integration  or  In¬ 
dependent  programs.  The  ICD  is  completed  at  the 
beginning  of  the  analyses  for  a  mission  area,  the 
CDD  and  CPD  are  program/increment  specific 
and  reviewed  to  evaluate  necessary  system 
modifications  periodically. 

5.2.2  Initial  Capabilities  Document  (ICD) 

The  ICD  makes  the  case  to  establish  the  need  for  a 
material  approach  to  resolve  a  specific  capability 
gap.  The  ICD  supports  the  Analysis  of  Alterna¬ 
tives  (AoA),  development  of  the  Technology  De¬ 
velopment  Strategy  (TDS),  and  various  milestone 
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decisions.  The  ICD  defines  the  capability  gap  in 
terms  of  the  functional  area(s),  relevant  range  of 
military  operations,  time,  obstacles  to  overcome, 
and  key  attributes  with  appropriate  Measures  of 
Effectiveness  (MOE).  Once  approved,  the  ICD  is 
not  normally  updated.  Format  for  the  ICD  is  found 
in  CJCSM  3170.01A. 

5.2.3  Capability  Development  Document 
(CDD) 

The  CDD  outlines  an  affordable  increment  of  mili¬ 
tarily  useful  and  supportable  operational  capabil¬ 
ity  that  can  be  effectively  developed,  produced  or 
acquired,  deployed,  and  sustained.  Each  increment 
of  capability  (CDD)  will  have  its  own  set  of  at¬ 
tributes  and  associated  performance  values  with 
thresholds  and  objectives  established  by  the  spon¬ 


sor  (Service)  with  input  from  the  user.  The  CDD 
performance  attributers  apply  to  only  one  incre¬ 
ment  and  include  Key  Performance  Parameters 
(KPPs)  and  other  parameters  that  will  guide  the 
development,  demonstration,  and  testing  of  the  cur¬ 
rent  increment.  The  CDD  will  outline  the  overall 
strategy  to  develop  the  full  or  complete  capability 
by  describing  all  increments.  (Figure  5-2)  The  CDD 
will  be  approved  at  Milestone  B  and  not  normally 
updated.  The  format  is  found  in  CJCSM  3 170.01  A. 

5.2.4  Capability  Production  Document 
(CPD) 

The  CPD  addresses  the  production  attributes  and 
quantities  specific  to  a  single  increment  and  is 
finalized  by  the  sponsor  after  the  Design  Readi¬ 
ness  Review  (DRR),  when  projected  capabilities 
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Figure  5-2.  Requirements  Definition  Process 


of  the  increment  in  development  have  been  speci¬ 
fied  with  sufficient  accuracy  to  begin  production. 
Design  trades  during  System  Development  and 
Demonstration  (SDD)  should  have  led  to  the  fi¬ 
nal  production  design  needed  for  Milestone  C. 
Initial  Operational  Test  and  Evaluation  (IOT&E) 
will  normally  test  to  the  values  in  the  CPD.  The 
threshold  and  objective  values  from  the  CDD  are, 
therefore,  superseded  by  the  specific  production 
values  detailed  in  the  CPD.  Reduction  in  any  KPP 
threshold  value  will  require  re-assessment  of  the 
military  utility  of  the  reduced  capability.  The  CPD 
will  always  reference  the  originating  ICD.  The 
format  is  found  in  CJCSM  3 170.01  A. 

5.2.5  System  Threat  Assessment  (STA) 

An  STA  is  prepared,  starting  at  Milestone  B,  by 
the  DoD  Component  Intelligence  Command  or 
Agency,  and  for  Acquisition  Category  (ACAT)  ID 
programs,  and  are  validated  by  the  Defense  Intel¬ 


ligence  Agency  (DIA)  (DoD  Directive  (DoDD) 
5105.21,  Defense  Intelligence  Agency,  19  May 
1977,  cancelled).  The  STA,  for  Defense  Acquisi¬ 
tion  Board  (DAB)  programs,  will  contain  a  con¬ 
cise  description  of  the  projected  future  operational 
threat  environment,  the  system-specific  threat,  the 
reactive  threat  that  could  affect  program  decisions, 
and  when  appropriate,  the  results  of  interactive 
analysis  obtained  by  the  Service  PM  when  evalu¬ 
ating  the  program  against  the  threat.  Threat  pro¬ 
jections  start  at  the  Initial  Operational  Capability 
(IOC)  and  extend  over  the  following  ten  years. 
The  STA  provides  the  basis  for  the  test  design  of 
threat  scenarios  and  the  acquisition  of  appropriate 
threat  targets,  equipment,  or  surrogates.  It  provides 
threat  data  for  Development  Test  and  Evaluation 
(DT&E)  and  Operational  Test  and  Evaluation 
(OT&E).  Vulnerability  and  lethality  analyses  dur¬ 
ing  live  fire  testing  of  ACAT  I  and  II  systems  are 
contingent  on  valid  threat  descriptions.  A  summary 
of  the  STA  is  included  in  Part  1  of  the  TEMP. 
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5.3  PROGRAM  DECISION 
DOCUMENTATION 

5.3.1  Acquisition  Decision  Memorandum 
(ADM) 

Under  Secretary  of  Defense  for  Acquisition,  Tech¬ 
nology,  and  Logistics  (USD(AT&L))  decisions  at 
major  defense  ACAT  ID  milestones  are  recorded 
in  a  document  known  as  an  ADM.  The  ADM  docu¬ 
ments  an  USD(AT&L)  decision  on  program 
progress  and  on  the  Acquisition  Program  Baseline 
(APB)  at  milestones  and  decision  reviews.  In  con¬ 
junction  with  an  ADM,  and  its  included  exit  crite¬ 
ria  for  the  next  phase,  the  APB  is  a  primary  pro¬ 
gram  guidance  document  providing  KPP  thresh¬ 
olds/objectives  for  systems  cost,  schedule,  and 
performance. 

5.3.2  Analysis  of  Alternatives  (AoA) 

An  AoA  is  normally  prepared  by  a  DoD  Compo¬ 
nent  agency  (or  Principal  Staff  Assistant  (PSA)  for 
ACAT  IA  programs),  other  than  the  Program  Man¬ 
agement  Office  (PMO),  for  each  milestone  review 
beginning  at  Milestone  A.  The  AoA  aids  decision 
makers  by  examining  the  relative  advantages  and 
disadvantages  of  program  alternatives,  shows  the 
sensitivity  of  each  alternative  to  possible  changes 
in  key  assumptions,  and  provides  the  rationale  for 
each  option.  The  guidance  in  the  Defense  Acquisi¬ 
tion  Guidebook,  Chapter  4,  requires  a  clear  link¬ 
age  between  the  AoA,  system  requirements,  and 
system  evaluation  MOE. 

The  driving  factor  behind  this  linkage  is  the  decision 
maker’s  reluctance  to  accept  modeling  or  simulation 
projections  for  system  performance  in  the  future 
without  actual  test  data  that  validates  AoA  results. 

5.4  PROGRAM  MANAGEMENT 
DOCUMENTATION 

5.4.1  Acquisition  Strategy 

An  event-based  acquisition  strategy  must  be 


formulated  at  the  start  of  a  development  program. 
Event-driven  acquisition  strategy  explicitly  links 
program  decisions  to  demonstrated  accomplish¬ 
ments  in  development,  testing,  and  initial  produc¬ 
tion.  The  strategy  constitutes  a  broad  set  of  con¬ 
cepts  that  provide  direction  and  control  for  the 
overall  development  and  production  effort.  The 
acquisition  strategy  is  updated  at  each  milestone 
and  decision  review  using  a  Working-level  Inte¬ 
grated  Product  Team  (WIPT)  structure  through¬ 
out  the  life  of  a  program.  The  level  of  detail  re¬ 
flected  in  the  acquisition  strategy  can  be  expected 
to  increase  as  a  program  matures.  The  acquisition 
strategy  serves  as  a  tailored  conceptual  basis  for 
formulating  other  program  functional  plans  such 
as  the  TEMP. 

It  is  important  that  T&E  interests  be  represented 
as  the  acquisition  strategy  is  formulated  because 
the  acquisition  strategy  should: 

•  Provide  an  overview  of  the  T&E  planned  for 
the  program,  ensuring  that  adequate  T&E  is 
conducted  prior  to  the  production  decision; 

•  Discuss  plans  for  providing  adequate  quanti¬ 
ties  of  test  hardware; 

•  Describe  levels  of  concurrence  and  combined 
Development  Test/Operational  Test  (DT/OT). 

5.4.2  Baseline  Documentation 

The  APB  will  initially  be  developed  before  Mile¬ 
stone  B  or  program  initiation  and  revised  for  each 
subsequent  milestone.  Baseline  parameters  repre¬ 
sent  the  cost,  schedule,  and  performance  objec¬ 
tives  and  thresholds  for  the  system  in  a  produc¬ 
tion  configuration.  Each  baseline  influences  the 
T&E  activities  in  the  succeeding  phases.  MOEs 
or  Measures  of  Performance  (MOPs)  shall  be  used 
in  describing  needed  capabilities  early  in  a  pro¬ 
gram.  Guidance  on  the  formulation  of  baselines 
is  found  in  the  Defense  Acquisition  Guidebook. 
Performance  demonstrated  during  T&E  of  produc¬ 
tion  systems  must  meet  or  exceed  the  thresholds. 
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The  thresholds  establish  deviation  limits  (actual 
or  anticipated  breach  triggers  reports)  for  KPPs 
beyond  which  the  PM  may  not  trade  off  cost, 
schedule,  or  performance  without  authorization  by 
the  Milestone  Decision  Authority  (MDA).  Baseline 
and  test  documentation  must  reflect  the  same  ex¬ 
pectations  for  system  performance.  The  total  num¬ 
ber  of  performance  parameters  shall  be  the  mini¬ 
mum  number  needed  to  characterize  the  major 
drivers  of  operational  effectiveness  and  suitabil¬ 
ity,  schedule,  technical  progress,  and  cost  for  a 
production  system  intended  for  deployment.  The 
performance  parameters  may  not  completely 
define  operational  effectiveness  or  suitability. 

5.4.3  Acquisition  Logistics  Planning 

Supportability  analyses  are  a  composite  of  all  sup¬ 
port  considerations  necessary  to  ensure  the  effec¬ 
tive  and  economical  support  of  a  system  at  all  lev¬ 
els  of  maintenance  for  its  programmed  life  cycle. 
Support  concepts  describe  the  overall  logistic  sup¬ 
port  program  and  include  logistics  requirements, 
tasks,  and  milestones  for  the  current  and  succeed¬ 
ing  phases  of  the  program.  The  analyses  serve  as 
the  source  document  for  logistic  support  testing 
requirements. 

Guidelines  for  logistic  support  analyses  are  docu¬ 
mented  in  Military  Standard  (MIL-STD)-1388-1A, 
Logistics  Support  Analysis,  11  April  1983.  This 
standard  identifies  how  T&E  programs  should 
be  planned  to  serve  the  following  three  logistics 
supportability  objectives: 

(1)  Provide  measured  data  for  input  into  system- 
level  estimates  of  readiness,  operational  costs, 
and  logistics  support  resource  requirements; 

(2)  Expose  supportability  problems  so  they  can 
be  corrected  prior  to  deployment; 

(3)  Demonstrate  contractor  compliance  with 
quantitative  supportability  —  related  design 
requirements. 


Development  of  an  effective  T&E  program  requires 
close  coordination  of  efforts  among  all  system 
engineering  disciplines,  especially  those  involved 
in  logistics  support  analyses.  Areas  of  particular 
interest  include  Reliability,  Availability,  and  Main¬ 
tainability  (RAM),  Human  Systems  Integration 
(HSI),  Environmental  Safety  and  Occupational 
Health  (ESOH),  and  post-deployment  evaluations. 
The  support  analyses  should  be  drafted  shortly 
before  program  start  to  provide  a  skeletal  frame¬ 
work  for  Logistics  Support  Analysis  (LSA),  to 
identify  initial  logistics  testing  requirements  that 
can  be  used  as  input  to  the  TEMP,  and  to  provide 
test  feedback  to  support  Integrated  Logistics  Sup¬ 
port  (ILS)  development.  Test  resources  will  be  lim¬ 
ited  early  in  the  program. 

5.4.4  Specification 

The  system  specification  is  a  document  used  in 
development  and  procurement  which  describes  the 
technical  performance  requirements  for  items, 
materials,  and  services  including  the  procedures 
by  which  it  will  be  determined  that  the  require¬ 
ments  have  been  met.  Specification  evolves  over 
the  developmental  phases  of  the  program  with 
increasing  levels  of  detail:  system,  item  perfor¬ 
mance,  item  detail,  process,  and  material.  Section 
4  of  the  specification  identifies  what  procedures 
(inspection,  demonstration,  analysis,  and  test)  will 
be  used  to  verily  the  performance  parameters  listed 
in  Section  3.  Further  details  may  be  found  in  MIL- 
STD-961D,  DoD  Standard  Practice  for  Defense 
Specifications,  22  March  1995,  Military  Defense 
Specification  Standard  Practices  (incorporated 
portions  of  MEL-STD-490,  Specification  Practices) 
which  is  fully  exempt  from  the  MIL-STD-491D 
waiver  process  because  it  is  a  “Standard  Practice.” 

5.4.5  Work  Breakdown  Structure  (WBS) 

A  program  WBS  shall  be  established  that  provides 
a  framework  for  program  and  technical  planning, 
cost  estimating,  resource  allocations,  performance 
measurements,  and  status  reporting.  Program  offices 
shall  tailor  a  program  WBS  for  each  program  using 
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the  guidance  in  Military  Handbook  (MIL-HDBK)- 
881,  Work  Breakdown  Structure,  2  January  1998. 
Level  2  of  the  WBS  hierarchical  structure  ad¬ 
dresses  system  level  T&E  with  sub-levels  for 
DT&E  and  OT&E.  Additionally,  each  configura¬ 
tion  item  structure  includes  details  of  the  integra¬ 
tion  and  test  requirements. 

5.5  TEST  PROGRAM  DOCUMENTATION 

5.5.1  Test  and  Evaluation  Master  Plan 
(TEMP) 

An  evaluation  strategy  is  developed  by  the  early 
concept  team  that  describes  how  the  capabilities 
in  the  CDD  will  be  evaluated  once  the  system  is 
developed.  The  evaluation  strategy,  part  of  the 
TDS,  provides  the  foundation  for  development 
of  the  program  TEMP  at  the  milestone  support¬ 
ing  program  start.  The  TEMP  is  the  basic  plan¬ 
ning  document  for  T&E  related  to  a  DoD  system 


acquisition  (Figure  5-3).  It  is  prepared  by  the 
PMO  with  the  OT  information  provided  by  the 
Service  Operational  Test  Agency  (OTA).  It  is 
used  by  Office  of  the  Secretary  of  Defense 
(OSD)  and  the  Services  for  planning,  review¬ 
ing,  and  approving  T&E  programs  and  provides 
the  basis  and  authority  for  all  other  detailed  T&E 
planning  documents.  The  TEMP  identifies  Criti¬ 
cal  Technical  Parameters  (CTPs),  performance 
characteristics  (MOE)  and  Critical  Operational 
Issues  (COIs);  and  it  describes  the  objectives, 
responsibilities,  resources,  and  schedules  for  all 
completed  and  planned  T&E.  The  TEMP,  in  the 
Defense  Acquisition  Guidebook  specified  format, 
is  required  by  DoDI  5000.2  for  ACAT  I,  IA,  and 
designated  oversight  programs  (see  enclosure  5 
for  more  information  regarding  the  TEMP).  For¬ 
mat  is  at  Service  discretion  for  ACAT  II  and  III 
programs.  For  example,  the  Army  requires  that 
each  TEMP  contain  the  full  set  of  approved  Criti¬ 
cal  Operational  Issues  and  Criteria  (COIC),  with 
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associated  scope  and  rationale,  forming  the  ba¬ 
sis  for  planning  the  system’s  evaluations.  (Army 
Regulation  (AR)  73-1,  Test  and  Evaluation 
Policy,  7  January  2002.) 

5.5.2  Evaluation  Plan 

Evaluation  planning  is  usually  included  within  the 
test  plan.  Evaluation  planning  considers  the  evalu¬ 
ation  and  analysis  techniques  that  will  be  required 
once  the  test  data  has  been  collected  and  pro¬ 
cessed.  Evaluation  is  linked  closely  to  the  test 
design,  especially  the  statistical  models  on  which 
the  test  design  is  built. 

The  Army  requires  a  system  evaluation  plan  de¬ 
scribing  the  evaluation  approach  that  addresses  the 
technical  and  operational  aspects  necessary  to  ad¬ 
dress  the  system’s  operational  effectiveness,  suit¬ 
ability,  and  survivability. 

The  objective  of  the  Army’s  emphasis  on  evalua¬ 
tion  is  to  address  the  issues;  describe  the  evalua¬ 
tion  of  issues  which  require  data  from  sources  other 
than  test;  state  the  technical  or  operational  issues 
and  criteria;  identify  data  sources  (data  source  ma¬ 
trix);  state  the  approach  to  the  independent  evalua¬ 
tion;  and  specify  the  analytical  plan  and  identify 
program  constraints.  (Reference  58) 

Evaluation  plans  are  prepared  for  all  systems  in 
development  by  the  independent  evaluators  during 
concept  exploration  and  in  coordination  with  the 
system  developer.  The  Army  System  Evaluation 
Plan  (SEP)  compliments  the  TEMP  and  is  devel¬ 
oped  in  support  of  the  initial  TEMP.  It  identifies 
each  evaluation  issue  and  the  methodology  to  be 
used  to  assess  it  and  specifies  requirements  for  ex¬ 
change  of  information  between  the  development/ 
operational  testers  and  materiel  developers. 

5.5.3  Test  Design 

Test  designers  need  to  ensure  that  the  test  is  con¬ 
structed  to  provide  useful  information  in  all  areas/ 
aspects  that  will  lead  to  an  assessment  of  the 


system  performance.  For  example,  a  complicated, 
even  ingenious,  test  that  does  not  provide  the 
information  required  by  the  decision  makers  is,  in 
many  respects,  a  failed  endeavor.  Therefore,  part 
of  the  process  of  developing  a  test  concept  or  test 
design  (the  distinction  between  these  vary  from 
organization  to  organization)  should  be  to  con¬ 
sider  whether  the  test  will  provide  the  informa¬ 
tion  required  by  the  decision  makers.  In  other 
words,  “Are  we  testing  the  right  things  in  the  right 
way... and  are  our  evaluations  meaningful?” 

The  test  design  is  statistical  and  analytical  in  na¬ 
ture  and  should  perform  the  following  functions: 

(1)  Structure  and  organize  the  approach  to  test¬ 
ing  in  terms  of  specific  test  objectives; 

(2)  Identify  key  MOEs  and  MOPs; 

(3)  Identify  the  required  data  and  demonstrate 
how  the  data  will  be  gathered,  stored,  ana¬ 
lyzed,  and  used  to  evaluate  MOEs; 

(4)  Indicate  what  part  Modeling  and  Simulation 
(M&S)  will  play  in  meeting  test  objectives; 

(5)  Identify  the  number  and  type  of  test  events 
and  required  resources. 

The  test  design  may  serve  as  a  foundation  for  the 
more-detailed  test  plan  and  specifies  the  test  objec¬ 
tives,  events,  instrumentation,  methodology,  data 
requirements,  data  management  needs,  and  analy¬ 
sis  requirements. 

5.5.4  Test  Plan 

The  test  plan  is  the  vehicle  that  translates  a  test 
concept  and  statistical/analytical  test  design  into 
concrete  resources,  procedures,  and  responsibili¬ 
ties.  The  size  and  complexity  of  a  test  program 
and  its  associated  test  plan  are  determined  by  the 
nature  of  the  system  being  tested  and  the  type  of 
testing  that  is  to  be  accomplished.  Some  major 
weapons  systems  may  require  large  numbers  of 
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separate  tests  to  satisfy  test  objectives  and,  thus, 
require  a  multi-volume  test  plan;  other  testing  may 
be  well-defined  by  a  relatively  brief  test  plan.  The 
test  plan  also  provides  a  description  of  the  equip¬ 
ment  configuration  and  known  limitations  to  the 
scope  of  testing.  The  type  of  information  typically 
included  in  a  test  plan  is  shown  in  Table  5-1. 


PRELIMINARY  PAGES 

I.  TITLE  PAGE 
ii.  APPROVAL  PAGE 

Hi.  CONDITIONS  FOR  RELEASE  OFTECHNICAL 
DATA 

iv.  PREFACE  (ABSTRACT) 

v.  EXECUTIVE  SUMMARY 

vi.  TABLE  OF  CONTENTS* 

*THE  ACTUAL  NUMBER  OF  THESE  PAGES  WILL  BE 
DETERMINED  BYTHE  LENGTH  OF  PRELIMINARY 
ELEMENTS  (E.G.,  TABLE  OF  CONTENTS, TERMS  AND 
ABBREVIATIONS,  ETC.). 

MAIN  BODY 

1.  INTRODUCTION 

2.  BACKGROUND 

3.  TEST  ITEM  DESCRIPTION 

4.  OVERALLTEST  OBJECTIVES 

5.  CONSTRAINTS  AND  LIMITATIONS 

6.  TEST  RESOURCES 

7.  SAFETY  REQUIREMENTS 

8.  SECURITY  REQUIREMENTS 

9.  TEST  PROJECT  MANAGEMENT 

10.  TEST  PROCEDURES 

11.  TEST  REPORTING 

12.  LOGISTICS 

13.  ENVIRONMENTAL  PROTECTION 
ANNEXES 

A.  TEST  CONDITION  MATRIX 

B.  REQUIREMENTS  TRACEABILITY 

C.  TEST  INFORMATION  SHEETS 

D.  PARAMETERS  LIST 

E.  DATA  ANALYSIS  PLAN 

F.  INSTRUMENTATION  PLAN 

G.  LOGISTICS  SUPPORT  PLAN 
H-Z  AS  REQUIRED 

LIST  OF  ABBREVIATIONS 

DISTRIBUTION 


SOURCE:  AFFTC  Test  Plan  Preparation  Guide,  May  1994. 


Table  5-1.  Sample  Test  Plan  Contents 


5.5.5  Outline  Test  Plan/Resources  Plan 

The  Army’s  Outline  Test  Plan  (OTP)  and  Air 
Force’s  Test  Resources  Plan  (TRP)  are  essential 
test  planning  documents.  They  are  formal  resource 
documents  specifying  the  resources  required  to 
support  the  test.  Since  the  OTP  or  TRP  provide 
the  basis  for  fiscal  programming  and  coordinating 
the  necessary  resources,  it  is  important  that  these 
documents  be  developed  in  advance  and  kept  cur¬ 
rent  to  reflect  maturing  resource  requirements  as 
the  test  program  develops.  The  Navy  makes  ex¬ 
tensive  use  of  the  TEMP  to  document  T&E  re¬ 
source  requirements.  Each  Service  has  periodic 
meetings  designed  to  review  resource  require¬ 
ments  and  resolve  problems  with  test  support. 

5.5.6  Test  Reports 

5.5.6.1  Quick-Look  Reports 

Quick-look  analyses  are  expeditious  analyses  per¬ 
formed  during  testing  using  limited  amounts  of 
the  database.  Such  analyses  often  are  used  to  as¬ 
sist  in  managing  test  operations.  Quick-look  re¬ 
ports  are  used  occasionally  to  inform  higher  au¬ 
thorities  of  test  results.  Quick-look  reports  may 
have  associated  briefings  that  present  T&E  results 
and  substantiate  conclusions  or  recommendations. 
Quick-look  reports  may  be  generated  by  the  con¬ 
tractor  or  government  agency.  They  are  of  par¬ 
ticularly  critical  interest  for  high-visibility  systems 
that  may  be  experiencing  some  development  dif¬ 
ficulties.  Techniques  and  formats  should  be  deter¬ 
mined  before  the  start  of  testing.  They  may  be 
exercised  during  pretest  trials. 

5.5.6.2  Final  Test  Report 

The  final  test  report  disseminates  the  test  informa¬ 
tion  to  decision  authorities,  program  office  staff, 
and  the  acquisition  community.  It  provides  a  per¬ 
manent  record  of  the  execution  of  the  test  and  its 
results.  The  final  test  report  should  relate  the  test 
results  to  the  critical  issues  and  address  the  objec¬ 
tives  stated  in  the  test  design  and  test  plan.  A  final 
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test  report  may  be  separated  into  two  sections — a 
main  section  providing  the  essential  information 
about  test  methods  and  results,  and  a  second  sec¬ 
tion  consisting  of  supporting  appendices  to  pro¬ 
vide  details  and  supplemental  information.  Gen¬ 
erally,  the  following  topics  are  included  in  the  main 
body  of  the  report: 

(1)  Test  purpose 

(2)  Issues  and  objectives 

(3)  Method  of  accomplishment 

(4)  Results  (keyed  to  the  objectives  and  issues) 

(5)  Discussion,  conclusions  and  recommenda¬ 
tions. 

Appendices  of  the  final  test  report  may  address 
the  following  topics: 

(1)  Detailed  test  description 

(2)  Test  environment 

(3)  Test  organization  and  operation 

(4)  Instrumentation 

(5)  Data  collection  and  management 

(6)  Test  data 

(7)  Data  analysis 

(8)  M&S 

(9)  RAM  information 

(10)  Personnel 

(11)  Training 

(12)  Safety 


(13)  Security 

(14)  Funding 

(15)  Asset  Disposition. 

The  final  test  report  may  contain  an  evaluation 
and  analysis  of  the  results,  or  the  evaluation  may 
be  issued  separately.  The  analysis  tells  what  the 
results  are,  whereas  an  evaluation  tells  what  the 
results  mean.  The  evaluation  builds  on  the  analy¬ 
sis  and  generalizes  from  it,  showing  how  the  re¬ 
sults  apply  outside  the  test  arena.  It  shows  what 
the  implications  of  the  test  are  and  may  provide 
recommendations.  The  evaluation  may  make  use 
of  independent  analyses  of  all  or  part  of  the  data; 
it  may  employ  data  from  other  sources  and  may 
use  M&S  to  generalize  the  results  and  extrapolate 
to  other  conditions.  In  the  case  of  the  Army,  a 
separate  System  Evaluation  Report  is  prepared  by 
independent  evaluators  within  the  Army  Evalua¬ 
tion  Center  (AEC). 

5.6  OTHER  TEST-RELATED  STATUS 
REPORTS 

5.6.1  End  of  Test  Reports 

The  Services  are  required  by  DoDI  5000.2,  Op¬ 
eration  of  the  Defense  Acquisition  System,  12  May 
2003  (enclosure  5)  to  submit  to  OSD  T&E  of¬ 
fices  copies  of  their  formal  DT&E,  OT&E,  and 
Live  Fire  Test  and  Evaluation  (LFT&E)  reports 
that  are  prepared  at  the  end  of  each  period  of  test¬ 
ing  for  ACAT  I,  IA,  and  oversight  programs.  These 
reports  will  generally  be  submitted  in  a  timely 
manner  to  permit  OSD  review. 

5.6.2  Beyond  Low-Rate  Initial  Production 
(BLRIP)  Report 

Before  an  ACAT  I  or  Director,  Operational  Test 
and  Evaluation  (DOT&E)  designated  program 
can  proceed  beyond  Low  Rate  Initial  Produc¬ 
tion  (LRIP),  the  DOT&E  must  submit  a  BLRIP 
report  to  the  Secretary  of  Defense  and  the  Senate 
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and  House  of  Representatives  Committees  on 
Armed  Services,  National  Security,  and  Appro¬ 
priations.  This  report  addresses  whether  the 
OT&E  performed  was  adequate  and  whether  the 
IOT&E  results  confirm  that  the  items  or  compo¬ 
nents  tested  are  effective  and  suitable  for  use  in 
combat  by  typical  military  users.  The  report  may 
include  information  on  the  results  of  LFT&E  for 
applicable  major  systems. 


5.7  SUMMARY 

A  wide  range  of  documentation  is  available  to  the 
test  manager  and  should  be  used  to  develop  T&E 
programs  that  address  all  relevant  issues.  The  PM 
must  work  to  ensure  that  T&E  requirements  are 
considered  at  the  outset  when  the  acquisition  strat¬ 
egy  is  formulated.  The  PM  must  also  require  early, 
close  coordination  and  a  continuing  dialogue 
among  those  responsible  for  integration  of  func¬ 
tional  area  planning  and  the  TEMP. 


5-10 


TYPES  OF  TEST 
AND  EVALUATION 


6.1  INTRODUCTION 

This  chapter  provides  a  brief  introduction  to  De¬ 
velopment  Test  and  Evaluation  (DT&E)  and  Op¬ 
erational  Test  and  Evaluation  (OT&E)  —  two  prin¬ 
cipal  types  of  Test  and  Evaluation  (T&E);  it  also 
discusses  the  role  of  qualification  testing  as  a  sub¬ 
element  of  development  testing.  Other  important 
types  of  T&E  are  introduced.  They  include:  multi- 
Service  testing;  joint  T&E;  Live  Fire  Testing; 
Nuclear,  Biological,  and  Chemical  (NBC)  testing; 
and  Nuclear  Hardness  and  Survivability  (NH&S) 
testing.  As  Figure  6-1  illustrates,  DT&E  and 
OT&E  are  performed  throughout  the  acquisition 
process  and  identified  by  nomenclature  that  may 
change  with  the  phase  of  the  acquisition  cycle  in 
which  they  occur. 

6.2  DEVELOPMENT  TEST  AND 
EVALUATION  (DT&E) 

DT&E  is  T&E  conducted  throughout  the  acquisi¬ 
tion  process  to  assist  in  engineering  design  and 
development  and  to  verify  that  technical  perfor¬ 
mance  specifications  have  been  met.  The  DT&E 
is  planned  and  monitored  by  the  developing 
agency  and  is  normally  conducted  by  the  contrac¬ 
tor.  However,  the  development  agency  may  per¬ 
form  technical  compliance  tests  before  OT&E.  It 
includes  the  T&E  of  components,  subsystems, 
Preplanned  Product  Improvement  (P3I)  changes, 
hardware/software  integration  and  production 
qualification  testing.  It  encompasses  the  use  of 
models,  simulations,  test  beds,  and  prototypes  or 
full-scale  engineering  development  models  of  the 
system.  DT&E  may  involve  a  wide  degree  of  test 
complexity,  depending  upon  the  type  of  system 
or  test  article  under  development;  e.g.,  tests  of 


electronic  breadboards  or  brassboards,  compo¬ 
nents,  subsystems  or  experimental  prototypes. 

The  DT&E  may  support  the  system  design  pro¬ 
cess  through  an  iterative  Modeling  and  Simula¬ 
tion  (M&S),  such  as  Simulation,  Test,  and  Evalu¬ 
ation  Process  (STEP)  that  involves  both  contrac¬ 
tor  and  government  personnel.  Because  contrac¬ 
tor  testing  plays  a  pivotal  role  in  the  total  test  pro¬ 
gram,  it  is  important  the  contractor  establishes  an 
integrated  test  plan  early  to  ensure  that  the  scope 
of  the  contractor’s  test  program  satisfies  govern¬ 
ment  and  contractor  test  objectives. 

Anti-tamper  component  level  verification  testing 
shall  take  place  prior  to  production  as  a  function 
of  DT/OT.  Component-level  testing  shall  not  as¬ 
sess  the  strength  of  the  anti-tamper  mechanism 
provided,  but  instead  verify  that  anti-tamper  per¬ 
forms  as  specified  by  the  source  contractor  or 
government  agency.  {Defense  Acquisition  Guide¬ 
book,  Chapter  3,  Test  and  Evaluation) 

The  Program  Manager  (PM)  remains  responsible 
for  the  ultimate  success  of  the  overall  program. 
The  PM  and  the  test  specialists  on  the  PM’s  staff 
must  foster  an  environment  that  provides  the  con¬ 
tractor  with  sufficient  latitude  to  pursue  innova¬ 
tive  solutions  to  technical  problems  and,  at  the 
same  time,  provides  the  data  needed  to  make  ra¬ 
tional  trade-off  decisions  between  cost,  schedule, 
and  performance  as  the  program  progresses. 

6.2.1  Production  Qualification  Test  ( PQT) 

Qualification  testing  is  a  form  of  development  test¬ 
ing  that  verifies  the  design  and  manufacturing 
process.  PQTs  are  formal  contractual  tests  that 
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Figure  6-1  .Testing  During  Acquisition 


confirm  the  integrity  of  the  system  design  over 
the  operational  and  environmental  range  in  the 
specification.  These  tests  usually  use  production 
hardware  fabricated  to  the  proposed  production 
design  specifications  and  drawings.  Such  tests  in¬ 
clude  contractual  Reliability  and  Maintainability 
(R&M)  demonstration  tests  required  before  pro¬ 
duction  release.  PQT  must  be  completed  before 
Full  Rate  Production  (FRP)  in  accordance  with 
Department  of  Defense  Instruction  (DoDI)  5000.2, 
Operation  of  the  Defense  Acquisition  System,  12 
May  2003. 

PQTs  may  be  conducted  on  Low  Rate  Initial  Pro¬ 
duction  (LRIP)  items  to  ensure  the  maturity  of  the 
manufacturing  process,  equipment,  and  proce¬ 
dures.  These  tests  are  conducted  on  each  item  or 
a  sample  lot  taken  at  random  from  the  first  pro¬ 
duction  lot  and  are  repeated  if  the  process  or  de¬ 
sign  is  changed  significantly  or  a  second  or  alter¬ 
native  source  is  brought  on  line.  These  tests  are 
also  conducted  against  contractual  design  and  per¬ 
formance  requirements. 

6.3  OPERATIONAL  TEST  AND 
EVALUATION  (OT&E) 

6.3.1  The  Difference  Between  Development 
and  Operational  Testing 

Air  Force  Manual  55-43,  Management  of  Opera¬ 
tional  Test  and  Evaluation,  published  in  June 
1979,  once  contained  the  following  account  of  the 
first  OT&E;  this  anecdote  serves  as  an  excellent 
illustration  of  the  difference  between  development 
and  operational  testing: 

The  test  and  evaluation  of  aircraft  and  air 
weapon  systems  started  with  the  contract 
awarded  to  the  Wright  brothers  in  1908. 

This  contract  specified  a  craft  which  would 
lift  two  men  with  a  total  weight  of  350 
pounds,  carry  enough  fuel  for  a  flight  of 
125  miles,  and  fly  40  miles  per  hour  in 
still  air.  The  contract  also  required  that  test¬ 
ing  be  conducted  to  assure  this  capability. 


What  we  now  call  development  test  and 
evaluation  (DT&E)  was  satisfied  when  the 
Wright  brothers  (the  developer)  demon¬ 
strated  that  their  airplane  could  meet  those 
first  contract  specifications.  However,  no 
immediate  military  mission  had  been  con¬ 
ceived  for  the  Wright  Flyer.  It  was  shipped 
to  Fort  Sam  Houston,  Texas,  where  Cap¬ 
tain  Benjamin  D.  Foulois,  the  pilot,  had 
orders  to  “teach  himself  to  fly.”  He  had  to 
determine  the  airplane’s  performance,  how 
to  maintain  it,  and  the  kind  of  organiza¬ 
tion  that  would  use  it.  Cavalry  wagon 
masters  had  to  be  trained  as  airplane  me¬ 
chanics,  and  Captain  Foulois  was  his  own 
instructor  pilot. 

In  the  process,  Captain  Foulois  subjected 
the  Wright  Flyer  to  test  and  evaluation 
under  operational  conditions.  Foulois  soon 
discovered  operational  deficiencies.  For 
example,  there  was  no  seat  on  the  airplane. 
During  hard  landings,  Foulois’  130  pound 
frame  usually  parted  company  from  the 
airplane.  To  correct  the  problem,  Foulois 
bolted  an  iron  tractor  seat  to  the  airplane. 
The  seat  helped,  but  Foulois  still  toppled 
from  his  perch  on  occasion.  As  a  further 
improvement,  Foulois  looped  his  Sam 
Browne  belt  through  the  seat  and  strapped 
himself  in.  Ever  since  then,  contoured  seats 
and  safety  belts  —  a  product  of  this  earli¬ 
est  “operational”  test  and  evaluation  — 
have  been  part  of  the  military  airplane. 

Captain  Foulois’  experience  may  seem  humorous 
now,  but  it  dramatically  illustrates  the  need  for 
operational  testing.  It  also  shows  that  operational 
testing  has  been  going  on  for  a  long  time. 

As  shown  in  Table  6-1  where  development  test¬ 
ing  is  focused  on  meeting  detailed  technical  speci¬ 
fications,  the  OT  focuses  on  the  actual  function¬ 
ing  of  the  equipment  in  a  realistic  combat  envi¬ 
ronment  in  which  the  equipment  must  interact  with 
humans  and  peripheral  equipment.  While  DT&E 
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Table  6-1.  Differences  Between  DT&E  and  IOT&E 


DT&E  _ 

•  CONTROLLED  BY  PM 

•  ONE-ON-ONETESTS 

•  CONTROLLED  ENVIRONMENT 

•  CONTRACTOR  INVOLVEMENT 

•  TRAINED,  EXPERIENCED  OPERATORS 

•  PRECISE  PERFORMANCE  OBJECTIVES  AND 
THRESHOLD  MEASUREMENT 

•  TESTTO  SPECIFICATION 

•  DEVELOPMENTTEST  ARTICLE 

and  OT&E  are  separate  activities  and  are  con¬ 
ducted  by  different  test  communities,  the  commu¬ 
nities  must  interact  frequently  and  are  generally 
complementary.  The  DT&E  provides  a  view  of 
the  potential  to  reach  technical  objectives,  and 
OT&E  provides  an  assessment  of  the  system’s 
potential  to  satisfy  user  requirements. 

6.3.2  The  Purpose  of  OT&E 

Operational  Test  and  Evaluation  is  defined  in  Title 
10,  U.S.C.  139  and  2399: 

The  field  test,  under  realistic  combat  con¬ 
ditions,  of  any  item  of  (or  key  compo¬ 
nent  of)  weapons,  equipment,  or  muni¬ 
tions  for  the  purposes  of  determining  the 
effectiveness  and  suitability  of  the  weap¬ 
ons,  equipment,  or  munitions  for  use  in 
combat  by  typical  military  users;  and  the 
evaluation  of  the  results  of  such  test.  This 
term  does  not  include  an  operational  as¬ 
sessment  based  exclusively  on  computer 
modeling,  simulation,  or  an  analysis  of 
system  requirements,  engineering  propos¬ 
als,  design  specifications,  or  any  other 
information  contained  in  program 
documents. 


_ IOT&E _ 

CONTROLLED  BY  INDEPENDENT  AGENCY 
MANY-ON-MANY  TESTS 

REALISTIC/TACTICAL  ENVIRONMENT  WITH  OPERATIONAL 
SCENARIO 

RESTRICTED  SYSTEM  CONTRACTOR  INVOLVEMENT 

USERTROOPS  RECENTLYTRAINED  ON  EQUIPMENT 

PERFORMANCE  MEASUREMENT  OF  OPERATIONAL 
EFFECTIVENESS  AND  SUITABILITY 

TESTTO  REQUIREMENTS 

PRODUCTION  REPRESENTATIVE  TEST  ARTICLE 

Definitions  of  operational  effectiveness  and 
operational  suitability  are  listed  below: 

Operational  Effectiveness:  Measure  of  the  over¬ 
all  ability  of  a  system  to  accomplish  a  mission 
when  used  by  representative  personnel  in  the  en¬ 
vironment  planned  or  expected  for  operational  em¬ 
ployment  of  the  system  considering  organization, 
doctrine,  tactics,  supportability,  survivability,  vul¬ 
nerability,  and  threat.  (Chairman  of  the  Joint  Chiefs 
of  Staff  Instruction  (CJCSI)  3170.01D,  Joint  Ca¬ 
pabilities  Integration  and  Development  System 
(JCIDS),  12  March  2004.) 

Operational  Suitability:  The  degree  to  which  a 
system  can  be  placed  and  sustained  satisfactorily 
in  field  use  with  consideration  being  given  to 
availability,  compatibility,  transportability,  interop¬ 
erability,  reliability,  wartime  usage  rates,  maintain¬ 
ability,  safety,  human  factors,  habitability,  man¬ 
power,  logistics  supportability,  natural  environmen¬ 
tal  effects  and  impacts,  documentation,  and  train¬ 
ing  requirements.  (CJCSI  3 170.0 ID) 

In  each  of  the  Services,  operational  testing  is 
conducted  under  the  auspices  of  an  organization 
that  is  independent  of  the  development  agency,  in 
as  operationally  realistic  environments  as  possible, 
with  hostile  forces  representative  of  the  anticipated 
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threat  and  with  typical  users  operating  and  main¬ 
taining  the  system.  In  other  words,  OT&E  is  con¬ 
ducted  to  ensure  that  new  systems  meet  the  user’s 
requirements,  operate  satisfactorily,  and  are  sup¬ 
portable  under  actual  field  conditions.  The  major 
questions  addressed  in  OT&E  are  shown  in  Figure 
6-2. 

Operational  Assessment  (EOA,  OA):  An  evalu¬ 
ation  of  operational  effectiveness  and  operational 
suitability  made  by  an  independent  OT  activity, 
with  user  support  as  required,  on  other  than  pro¬ 
duction  systems.  The  focus  of  an  OA  is  on  sig¬ 
nificant  trends  noted  in  development  efforts,  pro¬ 
grammatic  voids,  risk  areas,  adequacy  of  require¬ 
ments,  and  the  ability  of  the  program  to  support 
adequate  operational  testing.  An  OA  may  be  con¬ 
ducted  at  any  time  using  technology  demonstra¬ 
tors,  prototypes,  mock-ups,  Engineering  Develop¬ 
ment  Models  (EDMs),  or  simulators,  but  will  not 


substitute  for  the  Initial  Operational  T&E  (IOT&E) 
necessary  to  support  FRP  decisions.  Normally 
conducted  prior  to  Milestone  C.  ( Glossary  of  De¬ 
fense  Acquisition  Acronyms  &  Terms ,  September 

2003.) 

The  OA  normally  takes  place  during  each  phase 
of  the  acquisition  process  prior  to  IOT&E.  They 
are  used  to  provide  an  early  assessment  of  poten¬ 
tial  operational  effectiveness  and  suitability  for  de¬ 
cision  makers  at  decision  points.  These  assessments 
attempt  to  project  the  system’s  potential  to  meet 
the  user’s  requirements.  Assessments  conducted 
early  in  the  program  development  process  may 
be  called  Early  Operational  Assessments  (EOAs). 
When  using  an  evolutionary  acquisition  strategy, 
an  OA  is  required  for  all  incremehts  following  the 
one  experiencing  an  IOT&E  before  the  increment 
is  released  to  the  user.  (DoDI  5000.2) 


Figure  6-2.  Sample  Hierarchy  of  Questions  Addressed  in  OT&E 
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6.3.3  Initial  Operational  Test  and  Evaluation 
(IOT&E) 

The  OT&E  performed  in  support  of  the  FRP  de¬ 
cision  is  generally  known  as  IOT&E.  The  Navy 
calls  this  event  OPEVAL  (Operational  Evalua¬ 
tion).  The  IOT&E  occurs  during  LRIP  and  must 
be  completed  before  the  Full  Rate  Production  De¬ 
cision  Review  (FRPDR).  More  than  one  IOT&E 
may  be  conducted  on  the  system  if  there  are  sys¬ 
tem  performance  problems  requiring  re-test,  the 
system  is  de-certified,  or  a  need  exists  to  test  in 
different  environments.  The  IOT&E  is  conducted 
on  a  production  or  production  representative  sys¬ 
tem  using  typical  operational  personnel  in  a  re¬ 
alistic  combat  scenario. 

6.3.4  Follow-on  Operational  Test  and 
Evaluation  (FOT&E) 

The  OT&E  that  may  be  necessary  after  the 
FRPDR  to  refine  the  estimates  made  during 
IOT&E,  to  evaluate  changes,  and  to  re-evaluate 
the  system  to  ensure  that  it  continues  to  meet  op¬ 
erational  needs  and  retains  its  effectiveness  in  a 
new  environment  or  against  a  new  threat.  ( Glos¬ 
sary  of  Defense  Acquisition  Acronyms  &  Terms, 
September  2003)  FOT&E  is  conducted  during 
fielding/deployment  and  operational  support,  and 
sometimes  may  be  divided  into  two  separate  ac¬ 
tivities.  Preliminary  FOT&E  is  normally  con¬ 
ducted  after  the  Initial  Operational  Capability 
(IOC)  is  attained  to  assess  full  system  capability. 
It  is  conducted  by  the  OTA  or  designated  orga¬ 
nization  to  verify  the  correction  of  deficiencies, 
if  required,  and  to  assess  system  training  and  lo¬ 
gistics  status  not  evaluated  during  IOT&E.  Sub¬ 
sequent  FOT&E  is  conducted  on  production 
items  throughout  the  life  of  a  system.  The  re¬ 
sults  are  used  to  refine  estimates  of  operational 
effectiveness  and  suitability;  to  update  training, 
tactics,  techniques  and  doctrine;  and  to  identify 
operational  deficiencies  and  evaluate  modifica¬ 
tions.  This  later  FOT&E  often  is  conducted  by 
the  operating  command. 


6.4  MULTI-SERVICE  TEST  AND 
EVALUATION 

Multi-Service  Test  and  Evaluation  is  T&E  con¬ 
ducted  by  two  or  more  DoD  Components  for 
systems  to  be  acquired  by  more  than  one  DoD 
component,  or  for  a  DoD  component’s  systems 
that  have  interfaces  with  equipment  of  another 
DoD  component.  ( Glossary  of  Defense  Acquisi¬ 
tion  Acronyms  &  Terms)  All  affected  Services 
and  their  respective  Operational  Test  Agencies 
(OTAs)  participate  in  planning,  conducting,  re¬ 
porting,  and  evaluating  the  multi-service  test  pro¬ 
gram.  One  Service  is  designated  the  lead  Ser¬ 
vice  and  is  responsible  for  the  management  of 
the  program.  The  lead  Service  is  charged  with 
the  preparation  and  coordination  of  a  single  re¬ 
port  that  reflects  the  system’s  operational  effec¬ 
tiveness  and  suitability  for  each  Service. 

The  management  challenge  in  a  joint  acquisition 
program  conducting  multi-Service  T&E  stems 
from  the  fact  that  the  items  undergoing  testing 
will  not  necessarily  be  used  by  each  of  the  Ser¬ 
vices  for  identical  purposes.  Differences  among 
the  Services  usually  exist  in  performance  crite¬ 
ria,  tactics,  doctrine,  configuration  of  armament 
or  electronics,  and  the  operating  environment.  As 
a  result,  a  deficiency  or  discrepancy  considered 
disqualifying  by  one  Service  is  not  necessarily 
disqualifying  for  all  Services.  It  is  incumbent 
upon  the  lead  Service  to  establish  a  discrepancy 
reporting  system  that  permits  each  participating 
Service  to  document  all  discrepancies  noted.  At 
the  conclusion  of  a  multi-Service  T&E,  each 
participating  OT&E  agency  prepares  an  Indepen¬ 
dent  Evaluation  Report  (IER)  in  its  own  format 
and  submits  that  report  through  its  normal  Ser¬ 
vice  channels.  The  lead  Service  OT&E  agency 
prepares  the  documentation  that  goes  forward  to 
the  Milestone  Decision  Authority  (MDA).  This 
documentation  is  coordinated  with  all  participat¬ 
ing  OT&E  agencies. 
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6.5  JOINT  TEST  AND  EVALUATION 
(JT&E) 

Joint  Test  and  Evaluation  (JT&E)  is  not  the  same 
as  Multi-Service  T&E.  JT&E  is  a  specific  program 
activity  sponsored  and  largely  funded  by  an  Office 
of  the  Secretary  of  Defense  (OSD)  (Director,  Op¬ 
erational  Test  and  Evaluation  (DOT&E)).  JT&E 
programs  are  not  acquisition  oriented  but  are  a 
means  of  examining  joint-Service  tactics  and  doc¬ 
trine.  Past  joint-test  programs  have  been  conducted 
to  provide  information  required  by  the  Congress, 
the  OSD,  the  commanders  of  the  Unified  Com¬ 
mands,  and  the  Services.  The  overarching  policies 
and  details  of  formulating  a  JT&E  program  are  set 
forth  in  DoD  Directive  (DoDD)  5010.41,  Joint  Test 
and  Evaluation  (JT&E)  Program,  23  February 
1998. 

Responsibility  for  management  of  the  JT&E  pro¬ 
gram  was  transferred  from  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Lo¬ 
gistics  (USD(AT&L))  DT&E  to  the  Office  of  the 
DOT&E  in  December  2002.  Each  year  nomina¬ 
tions  are  submitted  by  Combatant  Commands, 
Department  of  Defense  (DoD)  agencies,  and  Ser¬ 
vices  to  the  JT&E  Program  Office  within  DOT&E 
for  review  and  processing.  The  JT&E  Program 
Office  identifies  those  to  be  funded  for  a  feasibil¬ 
ity  study,  the  DOT&E  assigns  lead  Service  respon¬ 
sibility  for  a  Quick  Reaction  Test  (6-12  months) 
or  a  full  JT&E  (up  to  3  years),  and  a  Joint  Test 
Force  Office  is  formed  to  conduct  JT&E  activi¬ 
ties  consistent  with  the  chartered  objectives  from 
the  participating  Services.  The  lead  and  partici¬ 
pating  Services  provide  personnel,  infrastructure 
support,  and  test  resources  for  JT&E  activities. 

The  JT&E  program’s  primary  objectives  include: 
assess  Service  system  interoperability  in  joint  op¬ 
erations;  evaluate  joint  technical  and  operational 
concepts  and  recommend  improvements;  validate 
testing  methodologies  that  have  joint  applications; 
improve  M&S  validity  with  field  exercise  data; 
increase  joint  mission  capability,  using  quantita¬ 
tive  data  for  analysis;  provide  feedback  to  the 


acquisition  and  joint  operations  communities;  and, 
improve  joint  Tactics,  Techniques,  and  Procedures 
(TTP). 

Each  JT&E  typically  produces  one  or  more  test 
products  that  provide  continuing  benefits  to  DoD, 
such  as  improvements  in:  joint  warfighting  capa¬ 
bilities;  joint  tactics,  techniques,  and  procedures; 
joint  and  individual  Service  training  programs; 
new  testing  methodologies;  joint  application  of 
M&Ss  that  can  support  acquisition,  TTP,  and  Con¬ 
cept  of  Operations  (COOs)  development;  and,  Uni¬ 
versal  task  List  update  and  input. 

6.6  LIVE  FIRE  TESTING 

The  Live  Fire  Test  (LFT)  Program  was  mandated 
by  the  Congress  in  the  National  Defense  Authori¬ 
zation  Act  (NDAA)  for  Fiscal  1987  (Public  Law 
99-661)  passed  in  November  1986.  Specifically, 
this  law  stipulated  that  a  major  [Acquisition  Cat¬ 
egory  (ACAT)  I  and  II]  program  development 
may  not  proceed  Beyond  Low  Rate  Initial  Pro¬ 
duction  (BLRIP)  until  realistic  survivability  or  (in 
the  case  of  missiles  and  munitions)  lethality  test¬ 
ing  has  been  completed. 

In  1984,  before  the  passage  of  this  legislation,  the 
OSD  had  chartered  a  joint  test  program  designed  to 
address  similar  questions  relative  to  systems  already 
in  field  use.  This  program.  Joint  LFT,  was  initially 
divided  into  two  distinct  parts:  Armor/Anti-armor  and 
Aircraft.  The  program’s  objectives  are  to: 

•  Gather  empirical  data  on  the  vulnerability  of 
existing  U.S.  systems  to  Soviet  weapons; 

•  Gather  empirical  data  on  the  lethality  of  exist¬ 
ing  U.S.  weapons  against  Soviet  systems; 

•  Provide  insights  into  the  design  changes  nec¬ 
essary  to  reduce  vulnerabilities  and  improve 
lethalities  of  existing  U.S.  weapon  systems; 

•  Calibrate  current  vulnerability  and  lethality 
models. 
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The  legislated  LFT  Program  complements  the 
older  Joint  Live  Fire  (JLF)  Program.  While  the 
JLF  Program  was  designed  to  test  systems  that 
were  fielded  before  being  completely  tested,  the 
spirit  and  intent  of  the  LFT  legislation  is  to  avoid 
the  need  to  play  “catch-up.”  This  program  not  only 
requires  the  Services  to  test  their  weapons  sys¬ 
tems  as  early  as  possible  against  the  expected  com¬ 
bat  threat,  but  also  before  FRP  to  identify  design 
characteristics  that  cause  undue  combat  damage 
or  measure  munitions  lethality.  Remedies  for  de¬ 
ficiencies  can  entail  required  retrofits,  production 
stoppages  or  other  more  time-consuming  solutions. 
The  essential  feature  of  live  fire  testing  is  that  ap¬ 
propriate  threat  munitions  are  fired  against  a  ma¬ 
jor  U.S.  system  configured  for  combat  to  test  its 
vulnerability  and/or  that  a  major  U.S.  munitions 
or  missile  is  fired  against  a  threat  target  config¬ 
ured  for  combat  to  test  the  lethality  of  the  muni¬ 
tions  or  missile. 

Live  Fire  Test  and  Evaluation  (LFT&E)  Guidelines 
were  first  issued  by  the  Deputy  Director,  T&E  (Live 
Fire  Testing)  in  May  1987  to  supplement  DoD  Test 
and  Evaluation  Master  Plan  (TEMP)  guidelines  in 
areas  pertaining  to  live  fire  testing  (Reference  33). 
These  guidelines  encompass  all  Major  Defense 
Acquisition  Programs  (MDAPs)  and  define  LFT 
requirements.  In  1994  Public  Law  103-355  directed 
that  oversight  of  Live  Fire  Testing  be  moved  within 
DoD  to  the  DOT&E.  Guidelines  for  this  program 
are  now  found  in  DoDI  5000.2  and  the  Defense 
Acquisition  Guidebook. 

6.7  NUCLEAR,  BIOLOGICAL,  AND 
CHEMICAL  (NBC)  WEAPONS 
TESTING 

The  testing  of  NBC  weapons  is  highly  special¬ 
ized  and  regulated.  PMs  involved  in  these  areas 
are  advised  to  consult  authorities  within  their  chain 
of  command  for  the  specific  directives,  instruc¬ 
tions,  and  regulations  that  apply  to  their  individual 
situations.  Nuclear  weapons  tests  are  divided  into 
categories  in  which  the  responsibilities  of  the 
Department  of  Energy  (DOE),  the  Defense 


Nuclear  Agency  (DNA)  and  the  military  Services 
are  clearly  assigned.  The  DOE  is  responsible  for 
nuclear  warhead  technical  tests;  the  DNA  is  re¬ 
sponsible  for  nuclear  weapons  effects  tests.  The 
Services  are  responsible  for  the  testing  of  Service- 
developed  components  of  nuclear  subsystems.  All 
nuclear  tests  are  conducted  within  the  provisions 
of  the  Limited  Test  Ban  Treaty  (LTBT)  that  gen¬ 
erally  restricts  nuclear  detonations  to  the  under¬ 
ground  environment.  Nuclear  weapons  testing  re¬ 
quires  extensive  coordination  between  Service  and 
DOE  test  personnel  (Reference  17). 

Since  the  United  States  signed  and  ratified  the 
Geneva  Protocol  of  1925,  U.S.  policy  has  been 
never  to  be  the  first  to  use  lethal  chemical  weap¬ 
ons;  it  may,  however,  retaliate  with  chemical 
weapons  if  so  attacked.  With  the  signing  and  rati¬ 
fication  of  the  1972  Biological  and  Toxin  Weapon 
Convention,  the  United  States  formally  adopted 
the  position  that  it  would  not  employ  biological 
or  toxin  weapons  under  any  circumstances.  All 
such  weapons  were  reported  destroyed  in  the  early 
1970s  (Reference  14). 

Regarding  retaliatory  capability  against  chemical 
weapons,  the  Service  Secretaries  are  responsible 
for  ensuring  that  their  organizations  establish  re¬ 
quirements  and  determine  the  military  character¬ 
istics  of  chemical  deterrent  items  and  chemical 
defense  items.  The  Army  has  been  designated  the 
DoD  executive  agent  for  DoD  chemical  warfare, 
research,  development,  and  acquisition  programs 
(Reference  14). 

United  States  policy  on  chemical  warfare  seeks  to: 

•  Deter  the  use  of  chemical  warfare  weapons  by 
other  nations; 

•  Provide  the  capability  to  retaliate  if  deterrence 
fails; 

•  Achieve  the  early  termination  of  chemical  war¬ 
fare  at  the  lowest  possible  intensity  (Reference 
14). 
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In  addition  to  the  customary  development  tests 
(conducted  to  determine  if  a  weapon  meets  tech¬ 
nical  specifications)  and  OTs  (conducted  to  deter¬ 
mine  if  a  weapon  will  be  useful  in  combat),  chemi¬ 
cal  weapons  testing  involves  two  types  of  chemi¬ 
cal  tests — chemical  mixing  and  biotoxicity.  Chemi¬ 
cal-mixing  tests  are  conducted  to  obtain  informa¬ 
tion  on  the  binary  chemical  reaction.  Biotoxicity 
tests  are  performed  to  assess  the  potency  of  the 
agent  generated.  Chemical  weapons  testing,  of 
necessity,  relies  heavily  on  the  use  of  nontoxic 
stimulants,  since  such  substances  are  more  eco¬ 
nomical  and  less  hazardous  and  open-air  testing 
of  live  agents  has  been  restricted  since  1969 
(Reference  14). 

6.8  NUCLEAR  HARDNESS  AND 
SURVIVABILITY  TESTING 

Nuclear  hardness  is  a  quantitative  description  of 
the  physical  attributes  of  a  system  or  component 
that  will  allow  it  to  survive  in  a  given  nuclear 
environment.  Nuclear  survivability  is  the  capabil¬ 
ity  of  a  system  to  survive  in  a  nuclear  environ¬ 
ment  and  to  accomplish  a  mission.  DoD  policy 
requires  the  incorporation  of  nuclear  hardness  and 
survivability  features  in  the  design,  acquisition,  and 
operation  of  major  and  nonmajor  systems  that  must 
perform  critical  missions  in  nuclear  conflicts. 
Nuclear  hardness  levels  must  be  quantified  and 
validated  (Reference  15). 


The  T&E  techniques  used  to  assess  nuclear  hard¬ 
ness  and  survivability  include:  nuclear  testing,  physi¬ 
cal  testing  in  a  simulated  environment,  modeling, 
simulation,  and  analysis.  Although  nuclear  tests  pro¬ 
vide  a  high  degree  of  fidelity  and  valid  results  for 
survivability  evaluation,  they  are  not  practical  for 
most  systems  due  to  cost,  long  lead  times,  and  in¬ 
ternational  treaty  constraints.  Underground  testing 
is  available  only  on  a  prioritized  basis  for  critical 
equipment  and  components,  and  is  subject  to  a  fre¬ 
quently  changing  test  schedule.  Physical  testing 
provides  an  opportunity  to  observe  personnel  and 
equipment  in  a  simulated  nuclear  environment. 
Modeling,  simulation,  and  analysis  are  particularly 
useful  in  the  early  stages  of  development  to  pro¬ 
vide  early  projections  before  system  hardware  is 
available.  These  methods  are  also  used  to  furnish 
assessments  in  an  area  that,  because  of  safety  or 
testing  limitations,  cannot  be  directly  observed 
through  nuclear  or  physical  testing. 

6.9  SUMMARY 

T&E  is  a  spectrum  of  techniques  used  to  address 
questions  about  critical  performance  parameters 
during  system  development.  These  questions  may 
involve  several  issues  including:  technical  and 
survivability  (development  testing);  effectiveness 
and  suitability  (operational  testing);  those  affect¬ 
ing  more  than  one  Service  (multi-Service  and  joint 
testing);  vulnerability  and  lethality  (live  fire  test¬ 
ing),  nuclear  survivability;  or  the  use  of  other  than 
conventional  weapons  (i.e.,  NBC). 
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MODULE 


DEVELOPMENTAL 
TEST  AND  EVALUATION 


Material  acquisition  is  an  iterative  process  of  designing,  building, 
testing,  identifying  deficiencies,  fixing,  retesting  and  repeating. 
Developmental  Test  and  Evaluation  (DT&E)  is  an  important  aspect 
of  this  process.  The  DT&E  is  performed  in  the  factory,  laboratory 
and  on  the  proving  ground.  It  is  conducted  by  subcontractors,  as 
they  are  developing  the  components  and  subassembly;  the  prime 
contractor,  as  he/she  assembles  the  components  and  ensures  inte¬ 
gration  of  the  system;  and  by  the  government,  to  demonstrate  how 
well  the  weapon  system  meets  its  technical  and  operational 
requirements.  This  module  describes  development  testing  and  the 
various  types  of  activities  it  involves.  The  module  also  discusses 
how  development  testing  is  used  to  support  the  technical  review 
process. 


INTRODUCTION  TO  DEVELOPMENT 
TEST  AND  EVALUATION 


7.1  INTRODUCTION 

Development  Test  and  Evaluation  (DT&E)  is  the 
Test  and  Evaluation  (T&E)  conducted  to  demon¬ 
strate  that  the  engineering  design  and  development 
process  is  complete.  It  is  used  by  the  contractor  to 
reduce  risk,  validate,  and  qualify  the  design,  and 
ensure  that  the  product  is  ready  for  government  ac¬ 
ceptance.  The  DT&E  results  are  evaluated  to  ensure 
that  design  risks  have  been  minimized  and  the  sys¬ 
tem  will  meet  specifications.  The  results  are  also  used 
to  estimate  the  system’s  military  utility  when  it  is 
introduced  into  service.  DT&E  serves  a  critical  pur¬ 
pose  in  reducing  the  risks  of  development  by  testing 
selected  high-risk  components  or  subsystems.  Finally, 
DT&E  is  the  government  developing  agency  tool 
used  to  confirm  that  the  system  performs  as  techni¬ 
cally  specified  and  that  the  system  is  ready  for  field 
testing.  This  chapter  provides  a  general  discussion 
of  contractor  and  government  DT&E  activities, 
stresses  the  need  for  an  integrated  test  program,  de¬ 
scribes  some  special-purpose  Development  Tests 
(DTs)  and  discusses  several  factors  that  may  influ¬ 
ence  the  extent  and  scope  of  the  DT&E  program. 

7.2  DT&E  AND  THE  SYSTEM 
ACQUISITION  CYCLE 

As  illustrated  in  Figure  7-1,  DT&E  is  conducted 
throughout  the  system  life  cycle.  DT&E  may  be¬ 
gin  before  program  initiation  with  the  evaluation 
of  evolving  technology,  and  it  continues  after  the 
system  is  in  the  field. 

7.2.1  DT&E  Prior  to  Program  Initiation 

Prior  to  program  initiation,  modeling,  simulations, 
and  technology  feasibility  testing  is  conducted  to 


confirm  that  the  technology  considered  for  the 
proposed  weapon  development  is  the  most  ad¬ 
vanced  available  and  that  it  is  technically  feasible. 

7.2.2  DT&E  During  Concept  Refinement 
(CR)  and  Technology  Development  (TD) 

Development  testing  that  takes  place  is  conducted 
by  a  contractor  or  the  government  to  assist  in  select¬ 
ing  preferred  alternative  system  concepts,  technolo¬ 
gies,  and  designs.  The  testing  conducted  depends 
on  the  state  of  technical  maturity  of  the  test  article’s 
design.  Government  test  evaluators  participate  in  this 
testing  because  information  obtained  can  be  used  to 
support  a  Systems  Requirements  Review  (SRR), 
early  test  planning  in  the  Technology  Development 
Strategy  (TDS),  and  development  of  the  Capabili¬ 
ties  Development  Document  (CDD)  and  the  Request 
for  Proposal  (REP)  for  Milestone  B.  The  informa¬ 
tion  obtained  from  these  tests  may  also  be  used  to 
support  a  program  start  decision  by  the  Services  or 
the  Office  of  the  Secretary  of  Defense  (OSD). 

7.2.3  DT&E  During  System  Development 
and  Demonstration  (SDD) 

Development  testing  conducted  is  used  to  demon¬ 
strate  that:  technical  risk  areas  have  been  identified 
and  can  be  reduced  to  acceptable  levels;  the  best 
technical  approach  can  be  identified;  and,  from  this 
point  on,  engineering  efforts  will  be  required  rather 
than  experimental  efforts.  It  supports  the  decision 
review  that  considers  transition  from  prototype  de¬ 
sign  into  advanced  engineering  and  construction 
of  the  Engineering  Development  Model  (EDM). 
This  DT&E  includes  contractor/govemment  inte¬ 
grated  testing,  engineering  design  testing,  and 
advanced  development  verification  testing. 
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SOURCE:  Adapted  and  modified  from  “Solving  the  Risk  Equation  In  Transitioning  from  Development  to  Production,”  January  19, 1984. 


Figure  7-1 .  Contractor’s  Integrated  Test  Requirements 


Development  testing  during  systems  integration  is 
most  often  conducted  at  the  contractor’s  facility.  It 
is  conducted  on  components,  subsystems,  brass- 
board  configurations,  or  advanced  development 
prototypes  to  evaluate  the  potential  application  of 
technology  and  related  design  approaches  before 
system  demonstration.  Component  interface  prob¬ 
lems  and  equipment  performance  capabilities  are 
evaluated.  The  use  of  properly  validated  analysis, 
Modeling  and  Simulation  (M&S)  is  encouraged, 
especially  during  the  early  phases  to  assess  those 
areas  that,  for  safety  or  testing  capability  limitations, 
cannot  be  observed  directly  through  testing.  M&Ss 
can  provide  early  projections  of  systems  perfor¬ 
mance,  effectiveness,  and  suitability,  and  can  re¬ 
duce  testing  costs.  This  T&E  also  may  include  ini¬ 
tial  environmental  assessments. 


Army  testing  of  the  Advanced  Attack  Helicopter 
(AAH)  provides  an  example  of  the  type  of  activi¬ 
ties  that  occur  during  development  testing/techni¬ 
cal  testing.  The  early  DT&E  of  the  AAH  was  con¬ 
ducted  by  the  Army  Engineering  Flight  Activity 
(EFA).  The  test  was  conducted  in  conjunction  with 
an  Early  Operational  Assessment  (EOA),  and  can¬ 
didate  designs  were  flown  more  than  90  hours  to 
evaluate  flight  handling  qualities  and  aircraft  per¬ 
formance.  This  test  also  included  the  firing  of  the 
30  millimeter  cannon  and  the  2.75-inch  rockets.  Re¬ 
liability,  Availability,  and  Maintainability  (RAM) 
data  were  obtained  throughout  the  test  program; 
and  these  data,  along  with  RAM  data  provided  from 
early  contractor  testing,  became  a  part  of  the 
system’s  RAM  database.  After  evaluating  the  re¬ 
sults,  the  Army  selected  a  contractor  to  proceed  with 
the  next  development  phase  of  the  AAH. 
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DT&E  conducted  during  system  demonstration 
provides  the  final  technical  data  for  determining  a 
system’s  readiness  to  transition  into  either  Low 
Rate  Initial  Production  (LRIP).  It  is  conducted 
using  advanced  EDMs  and  is  characterized  by  en¬ 
gineering  and  scientific  approaches  under  con¬ 
trolled  conditions.  The  qualification  testing  pro¬ 
vides  quantitative  and  qualitative  data  for  use  in 
the  system’s  evaluation.  The  evaluation  results  are 
used  by  the  development  community  and  are  also 
provided  to  Service  and  OSD  decision  authorities. 
These  tests  measure  technical  performance  includ¬ 
ing:  effectiveness,  RAM,  compatibility,  interoper¬ 
ability,  safety,  and  supportability.  They  include  tests 
of  human  engineering  and  technical  aspects  of  the 
system.  Demonstrations  of  whether  engineering  is 
reasonably  complete  and  if  solutions  to  all  signifi¬ 
cant  design  problems  are  in  hand  are  also  included. 

7.2.4  DT&E  During  Production  and 
Deployment  (P&D) 

7.2.4. 1  Low  Rate  Initial  Production  (LRIP) 

DT&E  may  be  conducted  on  EDMs  or  LRIP  ar¬ 
ticles  as  a  prelude  to  certifying  the  system  ready 
for  Initial  Operational  Test  and  Evaluation 
(IOT&E).  Each  Service  has  different  and  specific 
processes  incorporated  in  the  certification  for 
IOT&E  documentation.  The  Navy  conducts  ad¬ 
ditional  DT&E  for  certification  called 
TECHEVAL  (Technical  Evaluation).  This  is  a 
DT&E  event  controlled  by  the  program  office  that 
is  conducted  in  a  more  operationally  realistic  test 
environment.  The  Air  Force  has  developed  a  guide 
with  a  structured  process  using  templates  to  assist 
the  Program  Manager  (PM)  in  assessing  the 
program’s  readiness  for  IOT&E. 

As  an  example  of  testing  done  during  this  phase, 
the  Army  AAH  was  flown  in  a  series  of  Engi¬ 
neering  Design  Tests  (EDTs).  The  EDT-1,  -2  and 
-4  were  flown  at  the  contractor’s  facility.  (The 
EDT-3  requirement  was  deleted  during  program 
restructuring.)  The  objectives  of  these  flight  tests 
were  to  evaluate  the  handling  characteristics  of 


the  aircraft,  check  significant  performance  param¬ 
eters,  and  confirm  the  correction  of  deficiencies 
noted  during  earlier  testing.  The  EDT-5  was  con¬ 
ducted  at  an  Army  test  facility,  Yuma  Proving 
Ground,  Arizona.  The  objectives  of  this  test  were 
the  same  as  earlier  EDTs;  however,  the  testers 
were  required  to  ensure  that  all  discrepancies  were 
resolved  before  the  aircraft  entered  operational  test¬ 
ing.  During  the  EDTs,  Operational  Test  (OT)  per¬ 
sonnel  were  completing  OT  design,  bringing  to¬ 
gether  test  resources  and  observing  the  DT&E 
tests.  Additionally,  OT  personnel  were  compiling 
test  data,  such  as  the  system  contractor’s  test  re¬ 
sults,  from  other  sources.  The  evolving  DT  re¬ 
sults  and  contractor  data  were  made  available  to 
the  Critical  Design  Review  (CDR)  members  to 
ensure  that  each  configuration  item  design  was 
essentially  completed.  The  Army  conducted  a 
Physical  Configuration  Audit  (PCA)  to  provide  a 
technical  examination  to  verify  that  each  item  “as 
built”  conformed  to  the  technical  documentation 
defining  that  item. 

7.2.4.2  Post  Full  Rate  Production  Decision 
Review  (FRPDR) 

Development  testing  may  be  necessary  after  the 
Full-Rate  Production  (FRP)  decision  is  made.  This 
testing  is  normally  tailored  to  verify  correction  of 
identified  design  problems  and  demonstrate  the  sys¬ 
tem  modification’s  readiness  for  production.  This 
testing  is  conducted  under  controlled  conditions  and 
provides  quantitative  and  qualitative  data.  This  test¬ 
ing  is  conducted  on  production  items  delivered  from 
either  the  pilot  or  initial  production  runs.  To  ensure 
that  the  items  are  produced  according  to  contract 
specification,  limited  quantity  production  sampling 
processes  are  used.  This  testing  determines  whether 
the  system  has  successfully  transitioned  from  engi¬ 
neering  development  prototype  to  production,  and 
whether  it  meets  design  specifications. 

7.2.5  Operations  and  Support  (O&S) 

The  DT,  which  occurs  soon  after  the  Initial  Oper¬ 
ating  Capability  (IOC)  or  initial  deployment, 
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assesses  the  deployed  system’s  operational  readi¬ 
ness  and  supportability.  It  ensures  that  all  deficien¬ 
cies  noted  during  previous  testing  have  been  cor¬ 
rected,  evaluates  proposed  Product  Improvements 
(Pis)  and  block  upgrades,  and  ensures  that  Inte¬ 
grated  Logistics  Support  (ILS)  is  complete.  It  also 
evaluates  the  resources  on  hand  and  determines  if 
the  plans  to  ensure  operational  phase  readiness  and 
support  objectives  are  sufficient  to  maintain  the 
system  for  the  remainder  of  its  acquisition  life 
cycle.  For  mature  systems,  DT&E  is  performed 
to  assist  in  modifying  the  system  to  help  meet  new 
threats,  add  new  technologies,  or  aid  in  extending 
service  life. 

Once  a  system  approaches  the  end  of  its  useful¬ 
ness,  the  DT  conducted  is  concerned  with  the 
monitoring  of  a  system’s  current  state  of  opera¬ 
tional  effectiveness,  suitability,  and  readiness  to 
determine  whether  major  upgrades  are  necessary 
or  deficiencies  warrant  consideration  of  a  new 
system  replacement.  Tests  are  normally  conducted 
by  the  operational  testing  community;  however, 
the  DT&E  community  may  be  required  to  assess 
the  technical  aspects  of  the  system. 

7.3  DT&E  RESPONSIBILITIES 

As  illustrated  in  Figure  7-1,  the  primary  participants 
in  testing  are  the  prime  contractor,  subcontractor. 
Service  materiel  developer  or  developing  agency 
and  observing  is  the  Operational  Test  and  Evalua¬ 
tion  (OT&E)  agency.  In  some  Services,  there  are 
also  independent  evaluation  organizations  that  as¬ 
sist  the  testing  organization  in  designing  and  evalu¬ 
ating  development  tests.  As  the  figure  shows,  sys¬ 
tem  development  testing  is  performed  principally 
by  contractors  during  the  early  development  stages 
of  the  acquisition  cycle  and  by  government  test/ 
evaluation  organizations  during  the  later  phases. 

Army  testing  of  the  AAH  illustrates  the  type  of 
DT  performed  by  contractors  and  the  relation¬ 
ship  of  this  type  of  testing  to  government  DT&E 
activities.  During  the  contractor  competitive  test¬ 
ing  of  the  Army  AAH,  prime  contractor  and 


subcontractor  testing  included  design  support  tests, 
testing  of  individual  components,  establishing  fa¬ 
tigue  limits,  and  bench  testing  of  dynamic  com¬ 
ponents  to  demonstrate  sufficient  structural  integ¬ 
rity  to  conduct  the  Army  competitive  flight  test 
program.  Complete  dynamic  system  testing  was 
conducted  utilizing  ground  test  vehicles.  Besides 
supporting  the  contractor’s  development  effort, 
these  tests  provided  information  for  the  Army  tech¬ 
nical  review  process  as  the  systems,  Preliminary 
Design  Reviews  (PDRs),  and  Critical  Design 
Reviews  (CDRs)  were  conducted.  Following 
successful  completion  of  the  ground  test  vehicle 
qualification  testing,  first  flights  were  conducted 
on  the  two  types  of  competing  helicopters.  Each 
aircraft  was  being  flown  300  hours  before  deliv¬ 
ery  of  two  of  each  competing  aircraft  to  the  Army. 
The  contractor  flight  testing  was  oriented  toward 
flight-envelope  development,  demonstration  of 
structural  integrity,  and  evaluation  and  verification 
of  aircraft  flight  handling  qualities.  Some  weap¬ 
ons  system  testing  was  conducted  during  this 
phase.  Government  testers  used  much  of  the 
contractor’s  testing  data  to  develop  the  test  data 
matrices  as  part  of  the  government’s  DT  and  OT 
planning  efforts.  The  use  of  contractor  test  data 
reduced  the  testing  required  by  the  government 
and  added  validity  to  the  systems  already  tested 
and  data  received  from  other  sources. 

7.3.1  Contractor  Testing 

Materiel  DT&E  are  an  iterative  process  in  which 
a  contractor  designs  hardware  and  software,  evalu¬ 
ates  performance,  makes  changes  as  necessary,  and 
retests  for  performance  and  technical  compliance. 
Contractor  testing  plays  a  primary  role  in  the  total 
test  program,  and  the  results  of  contractor  tests 
are  useful  to  the  government  evaluator  in  support¬ 
ing  government  test  objectives.  It  is  important  that 
government  evaluators,  as  appropriate,  oversee 
contractor  system  tests  and  use  test  data  from  them 
to  address  government  testing  issues.  It  is  not  un¬ 
common  for  contractor  testing  to  be  conducted  at 
government  test  facilities,  since  contractors  often 
do  not  have  the  required  specialized  facilities  (e.g.. 
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for  testing  hazardous  components  or  for  missile 
flight  tests).  This  enables  government  evaluators 
to  monitor  the  tests  more  readily  and  increases 
government  confidence  in  the  test  results. 

Normally,  a  RFP  requires  that  the  winning  con¬ 
tractor  submit  an  Integrated  Engineering  Design 
Test  Plan  within  a  short  period  after  contract  ini¬ 
tiation  for  coordination  with  government  test  agen¬ 
cies  and  approval.  This  test  plan  should  include 
testing  required  by  the  Statement  of  Work  (SOW), 
specifications,  and  testing  expected  as  part  of  the 
engineering  development  and  integration  process. 
When  approved,  the  contractor’s  test  program  au¬ 
tomatically  becomes  part  of  the  development 
agency’s  Integrated  Test  Plan  (ITP). 

If  the  contractor  has  misinterpreted  the  RFP  re¬ 
quirements  and  the  Integrated  Engineering  Design 
Test  Plan  does  not  satisfy  government  test  objec¬ 
tives,  the  iterative  process  of  amending  the 
contractor’s  test  program  begins.  This  iterative 
process  must  be  accomplished  within  limited 
bounds  so  the  contractor  can  meet  the  test  objec¬ 
tives  without  significant  effects  on  contract  cost, 
schedule,  or  scope. 

7.3.2  Government  Testing 

Government  testing  is  performed  to:  demonstrate 
how  well  the  materiel  system  meets  its  technical 
compliance  requirements,  provide  data  to  assess 
developmental  risk  for  decision-making,  verify  that 
the  technical  and  support  problems  identified  in 
previous  testing  have  been  corrected,  and  ensure 
that  all  critical  issues  to  be  resolved  by  testing  have 
been  adequately  considered.  All  previous  testing, 
from  the  contractor’s  bench  testing  through  devel¬ 
opment  agency  testing  of  representative  prototypes, 
is  considered  during  government  evaluation. 

Government  materiel  development  organizations 
include  major  materiel  acquisition  commands  and, 
in  some  cases,  operational  commands.  The  mate¬ 
riel  acquisition  commands  have  T&E  organizations 
that  conduct  government  development  testing.  In 


addition  to  monitoring  and  participating  in  con¬ 
tractor  testing,  these  organizations  conduct  devel¬ 
opment  testing  on  selected  high-concern  areas  to 
evaluate  the  adequacy  of  systems  engineering, 
design,  development,  and  performance  to  specifi¬ 
cations.  The  Program  Management  Office  (PMO) 
must  be  involved  in  all  stages  of  testing  that  these 
organizations  perform. 

In  turn,  the  materiel  development/T&E  agencies 
conduct  T&E  of  the  systems  in  the  development 
stage  to  ensure  they  meet  technical  and  operational 
requirements.  These  organizations  operate  govern¬ 
ment  proving  grounds,  test  facilities,  and  labs;  and 
they  must  be  responsive  to  the  needs  of  the  PM  by 
providing  test  facilities,  personnel,  and  data 
acquisition  services,  as  required. 

7.4  TEST  PROGRAM  INTEGRATION 

During  the  development  of  a  weapon  system,  there 
are  a  number  of  tests  conducted  by  subcontractors, 
the  prime  contractor  and  the  government.  To  en¬ 
sure  these  tests  are  properly  time-phased,  that  ad¬ 
equate  resources  are  available,  and  to  minimize  un¬ 
necessary  testing,  a  coordinated  test  program  must 
be  developed  and  followed.  The  Test  and  Evalua¬ 
tion  Master  Plan  (TEMP)  normally  does  not  pro¬ 
vide  a  sufficient  level  of  detail  concerning  contrac¬ 
tor  or  subcontractor  tests.  A  contractor  or  PMO  FTP 
must  also  be  developed  to  describe  these  tests.  The 
PM  is  responsible  for  coordinating  the  total  T&E 
program.  The  PM  performs  this  task  with  the  assis¬ 
tance  of  the  T&E  IPT  whose  members  are  assembled 
from  development  agency,  user,  technical  and  op¬ 
erational  T&E,  logistics,  and  training  organizations. 
The  PM  must  remain  active  in  all  aspects  of  testing 
including  planning,  funding,  resourcing,  execution, 
and  reporting.  The  PM  plays  an  important  role  as 
the  interface  between  the  contractor  and  the  govern¬ 
ment  testing  community.  Recent  emphasis  on  early 
T&E  highlights  a  need  for  early  government  tester 
involvement  in  contractor  testing.  For  example,  dur¬ 
ing  development  of  the  AAH  test,  it  was  found  that 
having  program  management  personnel  on  the  test 
sites  improved  test  continuity,  facilitated  the  flow  of 
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spare  and  repair  parts,  provided  a  method  of  moni¬ 
toring  contractor  performance,  and  kept  the  Service 
headquarters  informed  with  timely  status  reports. 

7.4.1  Integrated  Test  Plan  (ITP) 

The  ITP  is  used  to  record  the  individual  test  plans 
for  the  subcontractor,  prime  contractor,  and  gov¬ 
ernment.  The  prime  contractor  should  be  contrac¬ 
tually  responsible  for  the  preparation  and  updating 
of  the  ITP,  and  the  contractor  and  Service- 
developing  agency  should  ensure  that  it  remains 
current.  The  ITP  includes  all  developmental  tests 
that  will  be  performed  by  the  prime  contractor  and 
the  subcontractors  at  both  the  system  and  subsystem 
levels.  It  is  a  detailed,  working-level  document  that 
assists  in  identifying  risk  as  well  as  duplicative  or 
missing  test  activities.  A  well-maintained  ITP 
facilitates  the  most  efficient  use  of  test  resources. 

7.4.2  Single  Integrated  Test  Policy 

Most  Services  have  adopted  a  single  integrated 
contractor/govemment  test  policy,  thereby  reduc¬ 
ing  much  of  the  government  testing  requirements. 
This  policy  stresses  independent  government 
evaluation  and  permits  an  evaluator  to  monitor 
contractor  and  government  test  programs  and 
evaluate  the  system  from  an  independent  perspec¬ 
tive.  The  policy  stresses  the  use  of  data  from  all 
sources  for  system  evaluation. 

7.5  AREAS  OF  DT&E  FOCUS 

7.5.1  Life  Testing 

Life  testing  is  performed  to  assess  the  effects  of 
long-term  exposure  to  various  portions  of  the  an¬ 
ticipated  environment.  These  tests  are  used  to  en¬ 
sure  the  system  will  not  fail  prematurely  due  to 
metal  fatigue,  component  aging,  or  other  problems 
caused  by  long-term  exposure  to  environmental 
effects.  It  is  important  that  the  requirements  for 
life  testing  are  identified  early  and  integrated  into 
the  system  test  plan.  Life  tests  are  time-consuming 
and  costly;  therefore,  life  testing  requirements  and 


life  characteristics  must  be  carefully  analyzed  con¬ 
current  with  the  initial  test  design.  Aging  failure 
data  must  be  collected  early  and  analyzed  through¬ 
out  the  testing  cycle.  If  life  characteristics  are  ig¬ 
nored  until  results  of  the  test  are  available,  exten¬ 
sive  redesign  and  project  delays  may  be  required. 
Accelerated  life  testing  techniques  are  available 
and  may  be  used  whenever  applicable. 

7.5.2  Design  Evaluation/Verification  Testing 

Design  evaluation  and  verification  testing  is  con¬ 
ducted  by  the  contractor  and/or  the  development 
agency  with  the  primary  objective  of  influencing 
system  design.  Design  evaluation  is  fully  inte¬ 
grated  into  the  development  test  cycle.  Its  purposes 
are  to: 

(1)  Determine  if  critical  system  technical  char¬ 
acteristics  are  achievable; 

(2)  Provide  data  for  refining  and  making  the 
hardware  more  rugged  to  comply  with  tech¬ 
nical  specification  requirements; 

(3)  Eliminate  as  many  technical  and  design  risks 
as  possible  or  to  determine  the  extent  to 
which  they  are  manageable; 

(4)  Provide  for  evolution  of  design  and  verifi¬ 
cation  of  the  adequacy  of  design  changes; 

(5)  Provide  information  in  support  of  develop¬ 
ment  efforts; 

(6)  Ensure  components,  subsystems,  and  systems 
are  adequately  developed  before  beginning 
OTs. 

7.5.3  Design  Limit  Testing 

Design  Limit  Tests  (DLTs)  are  integrated  into  the 
test  program  to  ensure  the  system  will  provide  ad¬ 
equate  performance  when  operated  at  outer  per¬ 
formance  limits  and  when  exposed  to  environmen¬ 
tal  conditions  expected  at  the  extremes  of  the 
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operating  envelope.  The  tests  are  based  on  mis¬ 
sion  profile  data.  Care  must  be  taken  to  ensure  all 
systems  and  subsystems  are  exposed  to  the  worst- 
case  environments,  with  adjustments  made  because 
of  stress  amplification  factors  and  cooling  prob¬ 
lems.  Care  must  also  be  taken  to  ensure  that  the 
system  is  not  operated  beyond  the  specified  de¬ 
sign  limits;  for  example,  an  aircraft  component 
may  have  to  be  tested  at  temperature  extremes  from 
an  Arctic  environment  to  a  desert  environment. 

7.5.4  Reliability  Development  Testing  (RDT) 

RDT  or  Reliability  Growth  Testing  (RGT)  is  a 
planned  Test,  Analyze,  Fix,  and  Test  (TAFT)  pro¬ 
cess  in  which  development  items  are  tested  under 
actual  or  simulated  mission-profile  environments  to 
disclose  design  deficiencies  and  to  provide  engi¬ 
neering  information  on  failure  modes  and  mecha¬ 
nisms.  The  purpose  of  RDT  is  to  provide  a  basis 
for  early  incorporation  of  corrective  actions  and  veri¬ 
fication  of  their  effectiveness  in  improving  the  reli¬ 
ability  of  equipment.  RDT  is  conducted  under  con¬ 
trolled  conditions  with  simulated  operational  mis¬ 
sion  and  environmental  profiles  to  determine  de¬ 
sign  and  manufacturing  process  weaknesses.  The 
RDT  process  emphasizes  reliability  growth  rather 
than  a  numerical  measurement.  Reliability  growth 
during  RDT  is  the  result  of  an  iterative  design  pro¬ 
cess  because,  as  the  failures  occur,  the  problems 
are  identified,  solutions  proposed,  the  redesign  is 
accomplished,  and  the  RDT  continues.  A  substan¬ 
tial  reliability  growth  TAFT  effort  was  conducted 
on  the  F-18  DT&E  for  selected  avionics  and  me¬ 
chanical  systems.  Although  the  TAFT  effort  added 
$100  million  to  the  Research,  Development,  Test 
and  Evaluation  (RDT&E)  Program,  it  is  estimated 
that  many  times  that  amount  will  be  saved  through 
lower  operational  and  maintenance  costs  through¬ 
out  the  system’s  life. 

7.5.5  Reliability,  Availability,  and 
Maintainability  (RAM) 

The  RAM  requirements  are  assessed  during  all 
contractor  and  government  testing.  The  data  are 


collected  from  each  test  event  and  placed  in  a 
RAM  database,  which  is  managed  by  the  devel¬ 
opment  agency.  Contractor  and  government  DTs 
provide  a  measure  of  the  system’s  common  RAM 
performance  against  stated  specifications  in  a  con¬ 
trolled  environment.  The  primary  emphasis  of 
RAM  data  collection  during  the  DT  is  to  provide 
an  assessment  of  the  system  RAM  parameters 
growth  and  a  basis  for  assessing  the  consequences 
of  any  differences  anticipated  during  field  opera¬ 
tions.  Early  projections  of  RAM  are  important  to 
logistics  support  planners.  The  test  data  facilities 
determination  of  spares  quantities,  maintenance  pro¬ 
cedures  and  support  equipment. 

7.6  SYSTEM  DESIGN  FOR  TESTING 

Built-in  Test  (BIT),  Built-in-test  Equipment  (BITE) 
and  Automated  Test  Equipment  (ATE)  are  major 
areas  that  must  be  considered  from  the  start  of  the 
design  effort.  Design  for  testing  (Figure  7-2)  ad¬ 
dresses  the  need  to:  (1)  collect  data  during  the 
development  process  concerning  particular  perfor¬ 
mance  characteristics;  (2)  enable  efficient  and  eco¬ 
nomical  production  by  providing  ready  access  to, 
and  measurement  of,  appropriate  acceptance  pa¬ 
rameters;  and  (3)  enable  rapid  and  accurate  as¬ 
sessment  of  the  status  of  the  product  to  the  lowest 
repairable  element  when  deployed.  Many  hard¬ 
ware  systems  have  testing  circuits  designed  and 
built-in.  This  early  planning  by  design  engineers 
allows  easy  testing  for  fault  isolation  of  circuits, 
both  in  system  development  phases  and  during 
operational  testing  and  deployment.  There  are 
computer  chips  in  which  more  than  half  of  the 
circuits  are  for  test/circuit  check  functions.  This 
type  of  circuit  design  requires  early  planning  by 
the  PM  to  ensure  the  RFP  requirements  include 
the  requirement  for  designed/BIT  capability. 
Evaluation  of  these  BIT/BITE/ATE  systems  must 
be  included  in  the  test  program. 

7.7  IMPACT  OF  WARRANTIES  ON  T&E 

A  warranty  or  guarantee  is  a  commitment  pro¬ 
vided  by  a  supplier  to  deliver  a  product  that  meets 
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specified  standards  for  a  specified  time.  With  a 
properly  structured  warranty,  the  contractor  must 
meet  technical  and  operational  requirements.  If 
the  product  should  fail  during  that  warranty  pe¬ 
riod,  the  contractor  must  replace  or  make  repairs 
at  no  additional  cost  to  the  government.  The 
Defense  Appropriations  Act  of  1984  requires 
warranties  or  guarantees  on  all  weapon  systems 
procurement.  This  Act  makes  warranties  a  stan¬ 
dard  item  on  most  fixed-price  production  con¬ 
tracts.  Incentives  are  the  main  thrust  of  warran¬ 
ties,  and  the  government  will  perform  a  reliabil¬ 
ity  demonstration  test  on  the  system  to  determine 
these  incentives.  Although  warranties  have  fa¬ 
vorable  advantages  to  the  government  during  the 
early  years  of  the  contract,  warranties  do  not  af¬ 
fect  the  types  of  testing  performed  to  ensure  the 
system  meets  technical  specifications  and  satis¬ 
fies  operational  effectiveness  and  suitability  re¬ 
quirements.  Warranties  do,  however,  affect  the 
amount  of  testing  required  to  establish  reliability. 


Because  the  standard  item  is  warranted,  less 
emphasis  on  that  portion  of  the  item  can  allow 
for  additional  emphasis  on  other  aspects  of  the 
item  not  covered  under  the  warranty.  Further,  the 
government  may  tend  to  have  more  confidence 
in  contractor  test  results  and  may  be  able,  there¬ 
fore,  to  avoid  some  duplication  of  test  effort.  The 
warranty  essentially  shifts  the  burden  of  risk  from 
the  government  to  the  contractor.  Warranties  can 
significantly  increase  the  price  of  the  contract, 
especially  if  high-cost  components  are  involved. 

7.8  DT&E  OF  LIMITED 

PROCUREMENT  QUANTITY 
PROGRAMS 

Programs  that  involve  the  procurement  of  rela¬ 
tively  few  items,  such  as  satellites,  some  large 
missiles,  and  unique  intelligence  equipment,  typi¬ 
cally  over  an  extended  period,  are  normally  sub¬ 
jected  to  modified  DT&E.  Occasionally,  a  unique 


Figure  7-2.  Design  for  Testing  Procedures 
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test  approach  that  deviates  from  the  standard  tim¬ 
ing  and  reporting  schedule  will  be  used.  The 
DT&E  principle  of  iterative  testing  starting  with 
components,  subsystems,  prototypes,  and  first- 
production  models  of  the  system  is  normally  ap¬ 
plied  to  limited  procurements.  It  is  important  that 
DT&E  and  OT&E  organizations  work  together 
to  ensure  that  integrated  T&E  plans  are  adapted/ 
tailored  to  the  overall  acquisition  strategy. 


7.9  SUMMARY 

DT&E  is  an  iterative  process  of  designing,  build¬ 
ing,  testing,  identifying  deficiencies,  fixing,  retest¬ 
ing,  and  repeating.  It  is  performed  in  the  factory, 
laboratory,  and  on  the  proving  ground  by  the  con¬ 
tractors  and  the  government.  Contractor  and  gov¬ 
ernment  testing  is  combined  into  one  integrated 
test  program  and  conducted  to  determine  if  the 
performance  requirements  have  been  met  and  to 
provide  data  to  the  decision  authority. 
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DT&E  SUPPORT  OF  TECHNICAL  REVIEWS 
AND  MILESTONE  DECISIONS 


8.1  INTRODUCTION 

Throughout  the  acquisition  process.  Development 
Test  and  Evaluation  (DT&E)  is  oriented  toward  the 
demonstration  of  specifications  showing  the  com¬ 
pleteness  and  adequacy  of  systems  engineering,  de¬ 
sign,  development,  and  performance.  A  critical  pur¬ 
pose  of  DT&E  is  to  identify  the  risks  of  develop¬ 
ment  by  testing  and  evaluating  selected  high-risk 
components  or  subsystems.  DT&E  is  the  developer’s 
tool  to  show  that  the  system  performs  as  specified  or 
that  deficiencies  have  been  corrected  and  the  system 
is  ready  for  operational  testing  and  fielding  (Figure 
8-1).  The  DT&E  results  are  used  throughout  the 
Systems  Engineering  Process  (SEP)  to  provide  valu¬ 
able  data  in  support  of  formal  design  reviews.  This 
chapter  describes  the  test’s  relationship  to  the  For¬ 
mal  Design  Reviews  (FDRs)  essential  to  the  SEP. 


8.2  DT&E  AND  THE  REVIEW  PROCESS 

8.2.1  The  Technical  Review  Process 

Technical  reviews  and  audits  are  conducted  by  the 
government  and  the  contractor  as  part  of  the  SEP 
to  ensure  the  design  meets  the  system,  subsystem, 
and  software  specifications.  Each  review  is  unique 
in  its  timing  and  orientation.  Some  reviews  build 
on  previous  reviews  and  take  the  design  and  test¬ 
ing  effort  one  step  closer  to  the  final  system  design 
to  satisfy  the  operational  concept/purpose  for  the 
weapon  system.  Table  8-1  illustrates  the  sequenc¬ 
ing  of  the  technical  reviews  in  relation  to  the  Test 
and  Evaluation  (T&E)  phases. 

The  review  process  was  established  to  ensure  that 
the  system  under  development  would  meet 


Figure  8-1.  Relationship  of  DT&E  to  the  Acquisition  Process 
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government  requirements.  The  reviews  evaluate 
data  from  contractor  and  government  testing,  en¬ 
gineering  analysis,  and  models  to  determine  if  the 
system  or  its  components  will  eventually  meet  all 
functional  and  physical  specifications  and  to  de¬ 
termine  the  final  system  design.  The  system  speci¬ 
fication  is  very  important  in  this  process.  It  is  the 
document  used  as  a  benchmark  to  compare  con¬ 
tractor  progress  in  designing  and  developing  the 
desired  product.  Guidelines  for  these  formal  tech¬ 
nical  reviews  and  audits  can  be  found  in  EIA  Stan¬ 
dard  632,  Processes  for  Engineering  a  System,  or 
IEEE  1220-1994,  Application  and  Management 
of  the  Systems  Engineering  Process. 

8.2.2  Testing  in  Support  of  Technical 
Reviews 

The  testing  community  must  be  continually  in¬ 
volved  in  supporting  the  technical  reviews  of  their 
systems.  The  timing  of  these  events  will  vary 
somewhat  from  program  to  program,  based  upon 
the  explicit  and  unique  needs  of  the  acquisition 


strategy.  Lower  level  design  reviews  will  lead  to 
system-level  reviews.  Decisions  made  at  these  re¬ 
views  have  major  impacts  on  the  system  test  de¬ 
sign,  resources  required  to  test,  and  the  develop¬ 
ment  of  the  Test  and  Evaluation  Master  Plan 
(TEMP)  and  other  documentation.  A  more  detailed 
discussion  of  testing  to  support  the  technical  re¬ 
views  is  provided  in  the  Systems  Engineering 
Fundamentals  Guide,  January  2001.  The  reviews 
focus  primarily  on  government  technical  specifi¬ 
cations  for  the  system.  Figure  8-2  illustrates  the 
program  specifications  and  how  they  are  devel¬ 
oped  in  the  system  life  cycle. 

8.2.3  Design  Reviews  and  Audits 

8.2.3.1  Concept  Refinement  (CR)  and 
Technology  Development  (TD) 

The  Alternative  Systems  Review  (ASR)  is  con¬ 
ducted  to  demonstrate  the  preferred  system 
concept(s).  Technical  reviews  conducted  during 
the  early  stages  of  development  tend  to  focus  on 
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top-level  concepts  and  system  definitions  clarifying 
the  requirements  on  which  subsequent  design  and 
development  activities  will  be  based. 

8.2.3.2  System  Development  and 
Demonstration  (SDD) 

The  System  Requirements  Review  (SRR)  is  nor¬ 
mally  conducted  late  in  the  system  concept  evalu¬ 
ation  or  shortly  after  program  initiation.  It  con¬ 
sists  of  a  review  of  the  system/system  segment 
specifications,  previously  known  as  the  “A” 
specifications  (System  Functional  Block  Dia¬ 
gram,  Reference  29,  Chapter  12),  and  is  con¬ 
ducted  after  the  accomplishment  of  functional 
analysis  and  preliminary  requirements  allocation. 
During  this  review,  the  systems  engineering  man¬ 
agement  activity  and  its  output  are  reviewed  for 
responsiveness  to  the  Statement  of  Work  (SOW) 
requirements.  The  primary  function  of  the  SRR 
is  to  ensure  that  system’s  requirements  have  been 
completed  and  properly  identified  and  that  there 
is  a  mutual  understanding  between  the  contrac¬ 
tor  and  the  government.  During  the  review,  the 
contractor  describes  his  progress  and  any  prob¬ 
lems  in  risk  identification  and  ranking,  risk 
avoidance  and  reduction,  trade-off  analysis, 
producibility  and  manufacturing  considerations, 
and  hazards  considerations.  The  results  of  inte¬ 
grated  test  planning  are  reviewed  to  ensure  the 
adequacy  of  planning  to  assess  the  design  and 
to  identify  risks.  Issues  of  testability  of  require¬ 
ments  should  be  discussed. 

The  System  Functional  Review  (SFR)  is  con¬ 
ducted  as  a  final  review  before  submittal  of  the 
prototype  design  products.  The  system  specifica¬ 
tion  is  validated  to  ensure  that  the  most  current 
specification  is  included  in  the  System  Functional 
Baseline  and  that  they  are  adequate  and  cost- 
effective  to  satisfy  validated  mission  requirements. 
The  SFR  encompasses  the  total  system  require¬ 
ment  of  operations,  maintenance,  test,  training, 
computers,  facilities,  personnel,  and  logistics  con¬ 
siderations.  A  technical  understanding  should  be 
reached  on  the  validity  and  the  degree  of 


completeness  of  specifications,  design,  Operational 
Concept  Documentation  (OCD),  Software  Re¬ 
quirements  Specifications  (SRSs),  and  Interface 
Requirements  Specifications  (IRSs)  during  this 
review. 

The  Software  Specification  Review  (SSR)  is  a 
formal  review  of  the  Computer  System  Configu¬ 
ration  Item  (CSCI)  requirements,  normally  held 
after  a  SFR  but  before  the  start  of  a  CSCI  pre¬ 
liminary  design.  Its  purpose  is  to  validate  the  al¬ 
located  baseline  for  preliminary  CSCI  design  by 
demonstrating  to  the  government  the  adequacy 
of  the  SRSs,  IRSs,  and  OCDs. 

The  Preliminary  Design  Review  (PDR)  is  a  for¬ 
mal  technical  review  of  the  basic  approach  for  a 
configuration  item.  It  is  conducted  at  the  Con¬ 
figuration  Item  (Cl)  and  system  level  early  in  the 
SDD  Phase  to  confirm  that  the  preliminary  de¬ 
sign  logically  follows  SFR  findings  and  meets 
the  system  requirements.  The  review  results  in 
an  approval  to  begin  the  detailed  design.  The 
draft  item  specifications  (performance)  are  re¬ 
viewed  during  the  PDR.  The  purpose  of  the 
PDR  is  to:  evaluate  the  progress,  technical  ad¬ 
equacy,  and  risk  resolution  (on  technical,  cost, 
and  schedule  basis)  of  the  Cl  design  approach; 
review  Development  Test  (DT)  and  Operational 
Test  (OT)  activities  to  measure  the  performance 
of  each  Cl;  and  establish  the  existence  and  com¬ 
patibility  of  the  physical  and  functional  interface 
among  the  Cl  and  other  equipment. 

The  Critical  Design  Review  (CDR)  may  be  con¬ 
ducted  on  each  Cl  and/or  at  the  system  level.  It  is 
conducted  on  the  Engineering  Development 
Model  (EDM)  design  when  the  detailed  design  is 
essentially  complete  and  prior  to  the  Functional 
Configuration  Audit  (FCA).  During  the  CDR,  the 
overall  technical  program  risks  associated  with 
each  Cl  are  also  reviewed  on  a  technical,  cost, 
and  schedule  basis.  It  includes  a  review  of  the 
item  specifications  (detail)  and  the  status  of  both 
the  system’s  hardware  and  software.  Input  from 
qualification  testing  should  assist  in  determination 
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of  readiness  for  design  freeze  and  Low  Rate  Ini¬ 
tial  Production  (LRIP).  Test  plans  are  reviewed 
to  assess  if  test  efforts  are  being  developed  suffi¬ 
ciently  to  indicate  a  Test  Readiness  Review  (TRR) 
would  be  successful.  The  CDR  should  precede 
the  program  DRR. 

The  TRR  was  a  formal  review  of  the  contractor’s 
readiness  to  begin  CSCI  testing,  but  has  evolved 
into  a  review  of  both  hardware  and  software.  It 
is  conducted  after  the  software  test  procedures 
are  available  and  computer  software  component 
testing  has  been  completed.  A  government  wit¬ 
ness  will  observe  the  system  demonstration  to 
verify  that  the  system  is  ready  to  proceed  with 
CSCI  testing.  The  purpose  of  the  TRR  is  for  the 
Program  Management  Office  (PMO)  to  deter¬ 
mine  whether  the  test  procedures  are  complete 
and  verify  their  compliance  with  test  plans  and 
descriptions. 

8.2.3.3  Production  and  Deployment  (P&D) 

The  FCA  is  a  formal  review  to  verify  that  the  CIs 
performance  complied  with  its  system  specifica¬ 
tion.  The  item  specifications  are  derived  from  the 
system  requirements  and  baseline  documentation. 
During  the  FCA,  all  relevant  test  data  is  reviewed 
to  verify  that  the  item  has  performed  as  required 
by  its  functional  and/or  allocated  configuration 
identification.  The  audit  is  conducted  on  the  item 
representative  (prototype  or  production)  of  the 
configuration  to  be  released  for  production.  The 
audit  consists  of  a  review  of  the  contractor’s  test 
procedures  and  results.  The  information  provided 
will  be  used  during  the  FCA  to  determine  the  sta¬ 
tus  of  planned  tests. 

The  Physical  Configuration  Audit  (PCA)  is  a  for¬ 
mal  review  which  establishes  the  product  baseline 
as  reflected  in  an  early  production  Cl.  It  is  the 
examination  of  the  as-built  version  of  hardware 
and  software  CIs  against  its  technical  documenta¬ 
tion.  The  PCA  also  determines  that  the  acceptance 
testing  requirements  prescribed  by  the  documen¬ 
tation  are  adequate  for  acceptance  of  production 


units  of  a  Cl  by  Quality  Assurance  (QA)  activi¬ 
ties.  It  includes  a  detailed  audit  of  engineering 
drawings,  final  Part  II  item  specifications  (detail), 
technical  data,  and  plans  for  testing  that  will  be 
utilized  during  production.  The  PCA  is  performed 
on  all  first  articles  and  on  the  first  CIs  delivered 
by  a  new  contractor. 

The  System  Verification  Review  (SVR)  is  a 
systems-level  configuration  audit  that  may  be  con¬ 
ducted  after  system  testing  is  completed.  The  ob¬ 
jective  is  to  verify  that  the  actual  performance  of 
the  Cl  (the  production  configuration),  as  deter¬ 
mined  through  testing,  complies  with  its  item 
specifications  (performance)  and  to  document  the 
results  of  the  qualification  tests.  The  SVR  and 
FCA  are  often  performed  at  the  same  time;  how¬ 
ever,  if  sufficient  test  results  are  not  available  at 
the  FCA  to  ensure  the  Cl  will  perform  in  its  op¬ 
erational  environment,  the  SVR  can  be  sched¬ 
uled  for  a  later  time. 

The  Production  Readiness  Review  (PRR)  is  an 
assessment  of  the  contractor’s  ability  to  produce 
the  items  on  the  contract.  It  is  usually  a  series  of 
reviews  conducted  before  an  LRIP  or  FRP  deci¬ 
sion.  For  more  information,  see  Chapter  10,  Pro¬ 
duction  Related  Testing  Activities. 

8.3  CONFIGURATION  CHANGE 
CONTROL 

Configuration  Change  Control  is  reviewed  to  as¬ 
sess  the  impact  of  engineering  or  design  changes. 
It  is  conducted  by  the  engineering,  T&E,  and 
Program  Manager  (PM)  portions  of  the  PMO. 
Most  approved  Class  I  Engineering  Change  Pro¬ 
posals  (ECPs)  will  require  additional  testing,  and 
the  test  manager  must  accommodate  the  new 
schedules  and  resource  requirements.  Adequate 
testing  must  be  accomplished  to  ensure  integra¬ 
tion  and  compatibility  of  these  changes.  For 
example,  an  Engineering  Change  Review  (ECR) 
was  conducted  to  replace  the  black  and  white 
monitors  and  integrate  color  monitors  into  the 
Airborne  Warning  and  Control  System 
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(AWACS).  Further,  the  AWACS  operating  soft¬ 
ware  had  to  be  upgraded  to  handle  color  enhance¬ 
ment.  The  review  was  conducted  by  the  govern¬ 
ment  PMO;  and  sections  of  the  PMO  were  tasked 
to  contract,  test,  engineer,  logistically  support, 
control,  cost,  and  finance  the  change  to  comple¬ 
tion.  Guidelines  for  configuration  control  and 
engineering  changes  are  discussed  in  EIA/IS-649 
(MIL-STD-480,  Configuration  Control  -  Engi¬ 
neering  Changes,  Deviations,  and  Waivers ). 


8.4  SUMMARY 

Design  reviews  are  an  integral  and  essential  part 
of  the  SEP.  The  meetings  range  from  very  formal 
reviews  by  government  and  contractor  PMs  to 
informal  technical  reviews  concerned  with  prod¬ 
uct  or  task  elements  of  the  Work  Breakdown 
Structure  (WBS).  Reviews  may  be  conducted  in 
increments  over  time.  All  reviews  share  the  com¬ 
mon  objective  of  determining  the  technical  ad¬ 
equacy  of  the  existing  design  to  meet  technical 
requirements.  The  DT/OT  assessments  and  test 
results  are  made  available  to  the  reviews,  and  it  is 
important  that  the  test  community  be  involved. 
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COMBINED  AND 
CONCURRENT  TESTING 


9.1  INTRODUCTION 

The  terms  “concurrency,”  “concurrent  testing,”  and 
“combined  testing”  are  sometimes  subject  to  mis¬ 
interpretation.  Concurrency  is  defined  as  an  ap¬ 
proach  to  system  development  and  acquisition  in 
which  phases  of  the  acquisition  process,  which 
normally  occur  sequentially,  overlap  to  some  ex¬ 
tent.  For  example,  a  weapon  system  enters  the 
production  phase  while  development  efforts  are 
still  underway. 

Concurrent  testing  refers  to  circumstances  when 
development  testing  and  operational  testing  take 
place  at  the  same  time  as  two  parallel,  but  sepa¬ 
rate  and  distinct,  activities.  In  contrast,  combined/ 
integrated  testing  refers  to  a  single  test  program 
conducted  to  support  Development  Test  (DT)  and 
Operational  Test  (OT)  objectives.  This  chapter  dis¬ 
cusses  the  use  of  combined  testing  and  concur¬ 
rent  testing,  and  highlights  some  of  the  advantages 
and  disadvantages  associated  with  these  ap¬ 
proaches.  (Table  9-1) 

9.2  COMBINING  DEVELOPMENT  TEST 
AND  OPERATIONAL  TEST 

Certain  test  events  can  be  organized  to  provide 
information  useful  to  development  testers  and  op¬ 
erational  testers.  For  example,  a  prototype  free- 
fall  munition  could  be  released  from  a  fighter  air¬ 
craft  at  operational  employment  conditions  instead 
of  from  a  static  stand  to  satisfy  DT  and  OT  objec¬ 
tives.  Such  instances  need  to  be  identified  to  pre¬ 
vent  unnecessary  duplication  of  effort  and  to  con¬ 
trol  costs.  A  combined  testing  approach  is  also 
appropriate  for  certain  specialized  types  of  test¬ 
ing.  For  example,  in  the  case  of  Nuclear  Hardness 


and  Survivability  (NHS)  testing,  systems  cannot 
be  tested  in  a  totally  realistic  operational  environ¬ 
ment;  therefore,  a  single  test  program  is  often  used 
to  meet  both  DT  and  OT  objectives. 

The  DoD  Instruction  (DoDI)  5000.2,  Operation 
of  the  Defense  Acquisition  System,  12  May  2003, 
encourages  combined/integrated  testing  suggest¬ 
ing  that  a  combined  Development  Test  and  Evalu¬ 
ation  (DT&E)  and  Operational  Test  and  Evalua¬ 
tion  (OT&E)  approach  should  be  considered  when 
there  are  time  and  cost  savings.  The  combined 
approach  must  not  compromise  either  DT  or  OT 
objectives.  If  this  approach  is  elected,  planning 
efforts  must  be  carefully  coordinated  early  in  the 
program  to  ensure  data  is  obtained  to  satisfy  the 
needs  of  both  the  developing  agency  and  the  in¬ 
dependent  operational  tester.  Care  must  also  be 
exercised  to  ensure  a  combined  test  program  con¬ 
tains  dedicated  OT  events  to  satisfy  the  require¬ 
ment  for  an  independent  evaluation.  A  final  inde¬ 
pendent  phase  of  OT&E  testing  shall  be  required 
for  Beyond  Low  Rate  Initial  Production  (BLRIP) 
decisions.  In  all  combined  test  programs,  provi¬ 
sions  for  separate  independent  development  and 
operational  evaluations  of  test  results  should  be 
provided. 

Service  regulations  describe  the  sequence  of  ac¬ 
tivities  in  a  combined  testing  program  as  follows: 

Although  OT&E  is  separate  and  distinct 
from  DT&E,  most  of  the  generated  data 
are  mutually  beneficial  and  freely  shared. 
Similarly,  the  resources  needed  to  conduct 
and  support  both  test  efforts  are  often  the 
same  or  very  similar.  Thus,  when  sequen¬ 
tial  DT&E  and  OT&E  efforts  would  cause 
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Table  9-1.  Combined  vs.  Concurrent  Testing:  Advantages  and  Limitations 


COMBINED  TESTING 

ADVANTAGES 

LIMITATIONS 

•  SHORTENS  TIME  REQUIRED  FOR  TESTING  AND, 

THUS,  THE  ACQUISITION  CYCLE. 

•  ACHIEVES  COST  SAVINGS  BY  ELIMINATING 

REDUNDANT  ACTIVITIES. 

•  EARLY  INVOLVEMENT  OF  OT&E  PERSONNEL  DURING 
SYSTEM  DEVELOPMENT  INCREASES  THEIR 

FAMILIARITY  WITH  SYSTEM. 

•  EARLY  INVOLVEMENT  OF  OT&E  PERSONNEL  PERMITS 
COMMUNICATION  OF  OPERATIONAL  CONCERNS  TO 
DEVELOPER  IN  TIME  TO  ALLOW  CHANGES  IN  SYSTEM 
DESIGN. 

•  REQUIRES  EXTENSIVE  EARLY  COORDINATION. 

•  TEST  OBJECTIVES  MAY  BE  COMPROMISED. 

•  REQUIRES  DEVELOPMENT  OF  DT/OT  COMMON  TEST 
DATABASE. 

•  COMBINED  TESTING  PROGRAMS  ARE  OFTEN  CON¬ 
DUCTED  IN  A  DEVELOPMENT  ENVIRONMENT. 

•  TEST  WILL  BE  DIFFICULT  TO  DESIGN  TO  MEET  DT  AND 
OT  REQUIREMENTS. 

•  THE  SYSTEM  CONTRACTOR  IS  PROHIBITED  BY  LAW 
FROM  PARTICIPATING  IN  IOT&E. 

•  TIME  CONSTRAINTS  MAY  RESULT  IN  LESS  COVERAGE 
THAN  PLANNED  FOR  OT&E  OBJECTIVES. 

CONCURRENT  TESTING 

ADVANTAGES 

LIMITATIONS 

•  SHORTENS  THE  TIME  REQUIRED  FOR  TESTING  AND, 
THUS,  THE  ACQUISITION  CYCLE. 

•  ACHIEVES  COST  SAVINGS  BY  OVERLAPPING 
REDUNDANT  ACTIVITIES. 

•  PROVIDES  EARLIER  FEEDBACK  TO  THE 

DEVELOPMENT  PROCESS. 

•  REQUIRES  EXTENSIVE  COORDINATION  OF  TEST 
ASSETS. 

•  IF  SYSTEM  DESIGN  IS  UNSTABLE  AND  FAR-REACHING 
MODIFICATIONSARE  MADE,  OT&E  MUST  BE  REPEATED. 

•  CONCURRENT  TESTING  PROGRAMS  OFTEN  DO  NOT 
HAVE  DT  DATAAVAILABLE  FOR  OT&E  PLANNING  AND 
EVALUATION. 

•  CONTRACTOR  PERSONNEL  FREQUENTLY  PERFORM 
MAINTENANCE  FUNCTIONS  IN  A  DT&E.  LOGISTIC 
SUPPORT  BY  USER  MUST  BE  AVAILABLE  EARLIER  FOR 
IOT&E. 

•  LIMITED  TEST  ASSETS  MAY  RESULT  IN  LESS  COVERAGE 
THAN  PLANNED  FOR  OT&E  OBJECTIVES. 

delay  or  increase  the  acquisition  cost  of 
the  system,  DT&E  and  OT&E  are  com¬ 
bined.  When  combined  testing  is  planned, 
the  necessary  test  conditions  and  data  re¬ 
quired  by  both  DT&E  and  OT&E  organi¬ 
zations  must  be  integrated.  Combined  test¬ 
ing  can  normally  be  divided  into  three 
segments. 

In  the  first  segment,  DT&E  event[s] 
usually  assume  priority  because  critical 


technical  and  engineering  tests  must  be 
accomplished  to  continue  the  engineering 
and  development  process.  During  this  early 
period,  OT&E  personnel  participate  to 
gain  familiarity  with  the  system  and  to  gain 
access  to  any  test  data  that  can  support 
OT&E.  Next,  the  combined  portion  of  the 
testing  frequently  includes  shared  objec¬ 
tives  or  joint  data  requirements.  The  last 
segment  normally  contains  the  dedicated 
OT&E  or  separate  OT&E  events  to  be 


9-2 


conducted  by  the  OT&E  agency.  The 
OT&E  agency  and  implementing  com¬ 
mand  must  ensure  the  combined  test  is 
planned  and  executed  to  provide  the  nec¬ 
essary  operational  test  information.  The 
OT&E  agency  provides  an  independent 
evaluation  of  the  OT&E  portion  and  is 
ultimately  responsible  for  achieving  OT&E 
objectives. 

The  testing  of  the  Navy’s  F-14  aircraft  has  been 
cited  as  an  example  of  a  successful  combined  Test 
and  Evaluation  (T&E)  program  (Reference  109). 
A  key  factor  in  the  success  of  the  F-14  approach 
was  the  selection  of  a  T&E  coordinator  respon¬ 
sible  for  supervising  the  generation  of  test  plans 
that  integrated  the  technical  requirements  of  the 
developers  with  the  operational  requirements  of 
the  users.  The  T&E  coordinator  was  also  respon¬ 
sible  for  the  allocation  of  test  resources  and  the 
overall  management  of  the  test.  In  a  paper  for  the 
Defense  Systems  Management  College,  Mr. 
Thomas  Hoivik  describes  the  successful  F-14  test 
program  as  follows: 

‘The  majority  of  the  Navy  developmental 
and  operational  testing  took  place  during 
the  same  period  and  even  on  the  same 
flights.  Maximum  use  was  made  of  con¬ 
tractor  demonstrations  witnessed  by  the 
Navy  testing  activities  to  obviate  the  re¬ 
testing  of  a  technical  point  already  dem¬ 
onstrated  by  the  contractor.  Witnessing  by 
testing  activities  was  crucially  important 
and  allowed  the  contractor’s  data  to  be 
readily  accepted  by  the  testing  activities. 
This  approach  also  helped  to  eliminate  re¬ 
dundancy  in  testing,  i.e.  the  testing  of  the 
same  performance  parameter  by  several 
different  activities  which  has  been  a  con¬ 
sistent  and  wasteful  feature  of  Navy  test¬ 
ing  in  the  past.” 

Obviously,  this  approach  placed  a  great  deal  of 
responsibility  directly  on  the  shoulders  of  the  T&E 
Coordinator,  and  required  the  T&E  Coordinator’s 


staff  to  deal  knowledgeably  with  a  wide-ranging 
and  complex  test  plan. 

9.3  CONCURRENT  TESTING 

In  1983,  a  senior  DoD  T&E  official  testified  that 
a  concurrent  testing  approach  is  usually  not  an 
effective  strategy  (Reference  104).  He  acknowl¬ 
edged,  however,  that  certain  test  events  may  pro¬ 
vide  information  useful  to  development  and 
operational  testers,  and  test  planners  must  be  alert 
to  identify  those  events.  His  testimony  included 
the  following  examples  of  situations  where  a 
concurrent  testing  approach  was  unsuccessful: 

(1)  During  A  AH  (Advanced  Attack  Helicop¬ 
ter)  testing  in  1981,  the  Target  Acquisi¬ 
tion  Designation  System  (TADS)  was  un¬ 
dergoing  developmental  and  operational 
testing  at  the  same  time.  The  schedule  did 
not  allow  enough  time  for  qualification 
testing  (a  development  test  activity)  of  the 
TADS  prototype  prior  to  a  full  field  test 
of  the  total  aircraft  system,  nor  was  there 
time  to  introduce  changes  to  TADS  prob¬ 
lems  discovered  in  tests.  As  a  result,  the 
TADS  performed  poorly  and  was  unreli¬ 
able  during  the  operational  test.  The  re¬ 
sulting  DS ARC  [Defense  Systems  Acqui¬ 
sition  Review  Council]  action  required  the 
Army  to  fix  and  retest  the  TADS  prior  to 
release  of  second  year  and  subsequent  pro¬ 
duction  funds. 

(2)  When  the  AIM-7  Sparrow  air-to-air  mis¬ 
sile  was  tested,  an  attempt  was  made  to 
move  into  operational  testing  while  devel¬ 
opmental  reliability  testing  was  still  under¬ 
way.  The  operational  test  was  suspended 
after  less  than  two  weeks  because  of  poor 
reliability  of  the  test  missiles.  The  program 
concentrated  on  an  intensive  reliability 
improvement  effort.  A  year  after  the  ini¬ 
tial  false  start,  a  full  operational  test  was 
conducted  and  completed  successfully. 
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(3)  The  Maverick  missile  had  a  similar  ex¬ 
perience  of  being  tested  in  an  operational 
environment  before  component  reliability 
testing  was  completed.  As  a  result, 
reliability  failures  had  a  major  impact  on 
the  operational  testers  and  resulted  in  the 
program  being  extended. 

9.4  ADVANTAGES  AND  LIMITATIONS 

Before  adopting  a  combined  or  concurrent  testing 
approach,  program  and  test  managers  are  advised 
to  consider  the  advantages  and  disadvantages  sum¬ 
marized  in  Table  9-1. 


9.5  SUMMARY 

A  combined  or  concurrent  testing  approach  may 
offer  an  effective  means  of  shortening  the  time 
required  for  testing  and  achieving  cost  savings.  If 
such  an  approach  is  used,  extensive  coordination 
is  required  to  ensure  the  development  and  opera¬ 
tional  requirements  are  addressed. 

It  is  possible  to  have  combined  test  teams,  con¬ 
sisting  of  DT&E,  OT&E,  and  contractor  person¬ 
nel  involved  throughout  the  testing  process.  The 
teams  can  provide  mutual  support  and  share  mu¬ 
tually  beneficial  data  as  long  as  the  test  program 
is  carefully  planned  and  executed  and  reporting 
activities  are  conducted  separately. 
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PRODUCTION-RELATED 
TESTING  ACTIVITIES 


10.1  INTRODUCTION 

Most  of  the  Test  and  Evaluation  (T&E)  discussed 
in  this  guidebook  concerns  the  testing  of  the  actual 
weapon  or  system  being  developed,  but  the  Pro¬ 
gram  Manager  (PM)  must  also  evaluate  production- 
related  test  activities  and  the  production  process. 
This  chapter  describes  production  management  and 
the  production  process  testing  required  to  ensure 
the  effectiveness  of  the  manufacturing  process  and 
the  producibility  of  the  system’s  design. 

Normally,  the  Development  Test  (DT)  and  Opera¬ 
tional  Test  (OT)  organizations  are  not  involved  di¬ 
rectly  in  this  process.  Usually,  the  manufacturing 
and  Quality  Assurance  (QA)  sections  of  the  pro¬ 
gram  office  and  a  representative  of  the  government 
Defense  Contract  Management  Agency  (DCMA) 
oversee/perform  many  of  these  functions. 

10.2  PRODUCTION  MANAGEMENT 

Production  (manufacturing)  management  is  the  ef¬ 
fective  use  of  resources  to  produce,  on  schedule, 
the  required  number  of  end  items  that  meet  speci¬ 
fied  quality,  performance,  and  cost.  Production 
management  includes,  but  is  not  limited  to,  indus¬ 
trial  resource  analysis,  producibility  assessment, 
producibility  engineering  and  planning,  produc¬ 
tion  engineering,  industrial  preparedness  planning, 
post-production  planning,  and  productivity  en¬ 
hancement.  Production  management  begins  early 
in  the  acquisition  process,  as  early  as  the  concept 
assessments,  and  is  specifically  addressed  at  each 
program  milestone  decision  points.  For  instance, 
before  program  initiation  production  feasibility, 
costs  and  risks  should  be  addressed.  The  PM  must 
conduct  an  Industrial  Resource  Analysis  (IRA)  to 


determine  the  availability  of  production  resources 
(e.g.,  capital,  material,  manpower)  required  to  sup¬ 
port  the  production  of  the  weapon  system.  On  the 
basis  of  the  results  of  the  IRA,  critical  materials, 
deficiencies  in  the  U.S.  industrial  base,  and  re¬ 
quirements  for  new  or  updated  manufacturing  tech¬ 
nology  can  be  identified.  Analysis  of  the  indus¬ 
trial-base  capacity  is  one  of  the  considerations  in 
preparing  for  the  program  start  decision.  As  de¬ 
velopment  proceeds,  the  manufacturing  strategy 
is  developed;  and  detailed  plans  are  made  for  pro¬ 
duction.  Independent  producibility  assessments, 
conducted  in  preparation  for  the  transition  from 
development  to  production,  are  reviewed  before 
entering  Low  Rate  Initial  Production  (LRIP). 
Once  production  starts,  the  producibility  of  the 
system  design  concept  is  evaluated  to  verify  that 
the  system  can  be  manufactured  in  compliance 
with  the  production-cost  and  the  industrial-base 
goals  and  thresholds. 

The  LRIP  decision  is  supported  by  an  assessment 
of  the  readiness  of  the  system  to  enter  production. 
The  system  cannot  enter  production  until  it  is  de¬ 
termined  the  principal  contractors  have  the  neces¬ 
sary  resources  (i.e.,  physical,  financial,  and  mana¬ 
gerial  capacity)  to  achieve  the  cost  and  schedule 
commitments  and  to  meet  peacetime  and  mobiliza¬ 
tion  requirements  for  production  of  the  system.  The 
method  of  assessing  production  readiness  is  the 
Production  Readiness  Review  (PRR),  which  is 
conducted  by  the  PM  and  staff. 

10.3  PRODUCTION  READINESS 
REVIEW  (PRR) 

The  following  are  guidelines  for  PRRs: 
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This  review  is  intended  to  determine  the 
status  of  completion  of  the  specific  actions 
which  must  be  satisfactorily  accomplished 
prior  to  executing  a  production  go-ahead 
decision.  The  review  is  accomplished  in 
an  incremental  fashion  before  commenc¬ 
ing  production.  Usually  two  initial  reviews 
and  one  final  review  are  conducted  to  as¬ 
sess  the  risk  in  exercising  the  production 
go-ahead  decision.  In  its  earlier  stages  the 
PRR  concerns  itself  with  gross  level  manu¬ 
facturing  concerns  such  as  the  need  for 
identifying  high-risk/low-yield  manufactur¬ 
ing  processes  or  materials  or  the  require¬ 
ment  for  manufacturing  development  ef¬ 
fort  to  satisfy  design  requirements.  Tim¬ 
ing  of  the  incremental  PRR’s  is  a  function 
of  program  posture  and  is  not  specifically 
locked  into  other  reviews. 

The  conduct  of  a  PRR  (Table  10-1)  is  the  respon¬ 
sibility  of  the  PM,  who  usually  appoints  a  direc¬ 
tor.  The  director  assembles  a  team  comprised  of 
individuals  in  the  disciplines  of  design,  industry, 
manufacturing,  procurement,  inventory  control, 
contracts,  engineering,  and  quality  training.  The 
PRR  director  organizes  and  manages  the  team 
effort  and  supervises  preparation  of  the  findings. 

10.4  QUALIFICATION  TESTING 

Qualification  testing  is  performed  to  verify  the 
design  and  manufacturing  process,  and  it  provides 
a  baseline  for  subsequent  acceptance  tests.  The 
production  qualification  testing  is  conducted  at  the 
unit,  subsystem,  and  system  level  on  production 
items  and  is  completed  before  the  production  de¬ 
cision.  The  results  of  these  tests  are  a  critical  fac¬ 
tor  in  assessing  the  system’s  readiness  for  produc¬ 
tion.  Down  line  Production  Qualification  Tests 
(PQTs)  are  performed  to  verify  process  control 
and  may  be  performed  on  selected  parameters 
rather  than  at  the  levels  originally  selected  for 
qualification. 


10.4.1  Production  Qualification  Tests  (PQTs) 

PQTs  are  a  series  of  formal  contractual  tests  con¬ 
ducted  to  ensure  design  integrity  over  the  speci¬ 
fied  operational  and  environmental  range.  The  tests 
are  conducted  on  pre-Full  Rate  Production  (FRP) 
items  fabricated  to  the  proposed  production  de¬ 
sign  drawings  and  specifications.  The  PQTs  in¬ 
clude  all  contractual  Reliability  and  Maintainabil¬ 
ity  (R&M)  demonstration  tests  required  prior  to 
production  release.  For  volume  acquisitions,  these 
tests  are  a  constraint  to  production  release. 

10.4.2  First  Article  Tests  (FATs) 

FATs  consist  of  a  series  of  formal  contractual  tests 
conducted  to  ensure  the  effectiveness  of  the  manu¬ 
facturing  process,  equipment,  and  procedures. 
These  tests  are  conducted  on  a  random  sample  from 
the  first  production  lot.  These  series  of  tests  are  re¬ 
peated  if  the  manufacturing  process,  equipment,  or 
procedure  is  changed  significantly  and  when  a  sec¬ 
ond  or  alternative  source  of  manufacturing  is 
brought  online.  [Federal  Acquisition  Regulation 
(FAR)  Part  9.3,  First  Article  Testing  and  Approval.] 

10.5  TRANSITION  TO  PRODUCTION 

In  an  acquisition  process,  often  the  first  indication 
that  a  system  will  experience  problems  is  during 
the  transition  from  engineering  design  to  LRIP. 
This  transition  continues  over  an  extended  period, 
often  months  or  years;  and  during  this  period,  the 
system  is  undergoing  stringent  contractor  and  gov¬ 
ernment  testing.  There  may  be  unexpected  fail¬ 
ures  requiring  significant  design  changes,  which 
impact  on  quality,  producibility,  supportability,  and 
may  require  program  schedule  slippage.  Long 
periods  of  transition  usually  indicate  that  insuffi¬ 
cient  attention  to  design  or  producibility  was  given 
early  in  the  program’s  acquisition  process. 

10.5.1  Transition  Planning 

Producibility  Engineering  and  Plan  (PEP)  is  the 
common  thread  that  guides  a  system  from  early 
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Table  1 0-1 .  PRR  Guidelines  Checklist 

PRODUCT  DESIGN 

•  PRODUCIBLE  AT  LOW  RISK 

•  STABILIZED  AT  LOW  RATE  OF  CHANGE 

•  VALIDATED 

•  RELIABILITY,  MAINTAINABILITY,  AND  PERFORMANCE  DEMONSTRATED 

•  COMPONENTS  ENGINEERING  HAS  APPROVED  ALL  PARTS  SELECTIONS 

INDUSTRIAL  RESOURCES 

•  ADEQUATE  PLANT  CAPACITY  (PEACETIME  AND  WARTIME  DEMANDS) 

•  FACILITIES,  SPECIAL  PRODUCTION  AND  TEST  EQUIPMENT,  AND  TOOLING  IDENTIFIED 

•  NEEDED  PLANT  MODERNIZATION  (CAD/CAM,  OTHER  AUTOMATION)  ACCOMPLISHED,  WHICH  PRODUCES 
AN  INVESTED  CAPTIVE  PAYBACK  IN  TWO  TO  FIVE  YEARS 

•  ASSOCIATED  COMPUTER  SOFTWARE  DEVELOPED 

•  SKILLED  PERSONNELAND  TRAINING  PROGRAMS  AVAILABLE 

PRODUCTION  ENGINEERING  AND  PLANNING 

•  PRODUCTION  PLAN  DEVELOPED  (REFERENCE  30) 

•  PRODUCTION  SCHEDULES  COMPATIBLE  WITH  DELIVERY  REQUIREMENTS 

•  MANUFACTURING  METHODS  AND  PROCESSES  INTEGRATED  WITH  FACILITIES,  EQUIPMENT,  TOOLING, 
AND  PLANT  LAYOUT 

•  VALUE  ENGINEERING  APPLIED 

•  ALTERNATE  PRODUCTION  APPROACHES  AVAILABLE 

•  DRAWINGS,  STANDARDS,  AND  SHOP  INSTRUCTIONS  ARE  EXPLICIT 

•  CONFIGURATION  MANAGEMENT  ADEQUATE 

•  PRODUCTION  POLICIES  AND  PROCEDURES  DOCUMENTED 

•  MANAGEMENT  INFORMATION  SYSTEM  ADEQUATE 

•  CONTRACTORS  MANAGEMENT  STRUCTURE  IS  ACCEPTABLE  TO  THE  PMO 

•  THE  PEP  CHECKLIST  HAS  BEEN  REVIEWED 

MATERIALS 

•  ALL  SELECTED  MATERIALS  APPROVED  BY  CONTRACTOR’S  MATERIAL  ENGINEERS 

•  BILL  OF  MATERIALS  PREPARED 

•  MAKE-OR-BUY  DECISIONS  COMPLETE 

•  PROCUREMENT  OF  LONG  LEAD-TIME  ITEMS  IDENTIFIED 

•  SOLE-SOURCE  AND  GOVERNMENT-FURNISHED  ITEMS  IDENTIFIED 

•  CONTRACTORS  INVENTORY-CONTROL  SYSTEM  ADEQUATE 

•  CONTRACTORS  MATERIAL  COST  PROCUREMENT  PLAN  COMPLETE 

QUALITY  ASSURANCE  (QA) 

•  QUALITY  PLAN  IN  ACCORDANCE  WITH  CONTRACT  REQUIREMENTS 

•  QUALITY  CONTROL  PROCEDURES  AND  ACCEPTANCE  CRITERIA  ESTABLISHED 

•  QA  ORGANIZATION  PARTICIPATES  IN  PRODUCTION  PLANNING  EFFORT 

LOGISTICS 

•  OPERATIONAL  SUPPORT,  TEST,  AND  DIAGNOSTIC  EQUIPMENT  AVAILABLE  AT  SYSTEM  DEPLOYMENT 

•  TRAINING  AIDS,  SIMULATORS,  AND  OTHER  DEVICES  READY  AT  SYSTEM  DEPLOYMENT 

•  SPARES  INTEGRATED  INTO  PRODUCTION  LOT  FLOW 


concept  to  production.  Planning  is  a  management 
tool  used  to  ensure  that  adequate  risk-handling 
measures  have  been  taken  to  transition  from  de¬ 
velopment  to  production.  It  contains  a  checklist 
to  be  used  during  the  readiness  reviews.  Planning 
should  tie  together  the  applications  of  designing, 
testing,  and  manufacturing  activities  to  reduce  data 
requirements,  duplication  of  effort,  costs  and 
scheduling,  and  to  ensure  early  success  of  the 
LRIP  first  production  article. 

10.5.2  Testing  During  the  Transition 

Testing  accomplished  during  the  transition  from 
development  to  production  will  include  acceptance 
testing,  manufacturing  screening,  and  final  test¬ 
ing.  These  technical  tests  are  performed  by  the 
contractor  to  ensure  the  system  will  transition 
smoothly  and  that  test  design  and  manufacturing 
issues  affecting  design  are  addressed.  During  this 
same  period,  the  government  will  be  using  the 
latest  available  configuration  item  to  conduct  the 
Initial  Operational  Test  and  Evaluation  (IOT&E). 
The  impact  of  these  tests  may  overwhelm  the  con¬ 
figuration  management  of  the  system  unless  care¬ 
ful  planning  is  accomplished  to  handle  these 
changes. 

10.6  LOW  RATE  INITIAL  PRODUCTION 
(LRIP) 

LRIP  is  the  production  of  a  system  in  limited 
quantity  to  provide  articles  for  IOT&E  and  to  dem¬ 
onstrate  production  capability.  Also,  it  permits  an 
orderly  increase  in  the  production  rate  sufficient 
to  lead  to  FRP  upon  successful  completion  of  op¬ 
erational  testing.  The  decision  to  have  an  LRIP  is 
made  at  the  Milestone  C  approval  of  the  program 
acquisition  strategy.  At  that  time,  the  PM  must 
identify  the  quantity  to  be  produced  during  LRIP 
and  validate  the  quantity  of  LRIP  articles  to  be 
used  for  IOT&E  [Acquisition  Category  (ACAT)  I 
approved  by  the  Director,  Operational  Test  and 
Evaluation  (DOT&E),  ACAT  II  and  III  approved 
by  Service  Operational  Test  Agency  (OTA)]  (De¬ 
partment  of  Defense  Instruction  (DoDI)  5000.2, 


Operation  of  the  Defense  Acquisition  System,  12 
May  2003).  When  the  decision  authority  thinks 
the  systems  will  not  perform  to  expectation,  the 
PM  may  direct  that  it  not  proceed  into  LRIP  until 
there  is  a  program  review.  The  DOT&E  submits 
a  Beyond  LRIP  (BLRIP)  report,  on  all  oversight 
systems,  to  congressional  committees  before  the 
FRP  decision  approving  the  system  to  proceed 
beyond  LRIP  is  made. 

10.7  PRODUCTION  ACCEPTANCE  TEST 
AND  EVALUATION  (PAT&E) 

The  PAT&E  ensures  that  production  items  dem¬ 
onstrate  the  fulfillment  of  the  requirements  and 
specifications  of  the  procuring  contract  or  agree¬ 
ments.  The  testing  also  ensures  the  system  being 
produced  demonstrates  the  same  performance  as 
the  pre-FRP  models.  The  procured  items  or  sys¬ 
tem  must  operate  in  accordance  with  system  and 
item  specifications.  The  PAT&E  is  usually  con¬ 
ducted  by  the  program  office  QA  section  at  the 
contractor’s  plant  and  may  involve  operational 
users. 

For  example,  for  the  Rockwell  B-1B  Bomber  pro¬ 
duction  acceptance,  Rockwell  and  Air  Force  QA 
inspectors  reviewed  all  manufacturing  and  ground 
testing  results  for  each  aircraft.  In  addition,  a  flight 
test  team,  composed  of  contractor  and  Air  Force 
test  pilots,  flew  each  aircraft  a  minimum  of  10 
hours,  demonstrating  all  on-board  aircraft  systems 
while  in  flight.  Any  discrepancies  in  flight  were 
noted,  corrected,  and  tested  on  the  ground;  they 
were  then  retested  on  subsequent  checkouts  and 
acceptance  flights.  Once  each  aircraft  had  passed 
all  tests  and  all  systems  were  fully  operational, 
Air  Force  authorities  accepted  the  aircraft.  The  test 
documentation  also  became  part  of  the  delivered 
package.  During  this  test  period,  the  program  of¬ 
fice  monitored  each  aircraft’s  daily  progress. 

10.8  SUMMARY 

A  primary  purpose  of  production-related  testing 
is  to  lower  the  production  risk  in  a  Major  Defense 
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Acquisition  Program  (MDAP).  The  PM  must  en¬ 
sure  the  contractor’s  manufacturing  strategy  and 
capabilities  will  result  in  the  desired  product 
within  acceptable  cost.  The  LRIP  and  PAT&E 
also  play  major  roles  in  ensuring  the  production 


unit  is  identical  to  the  design  drawings,  conforms 
to  the  specifications  of  the  contract,  that  the  IOT&E 
is  conducted  with  representative  system  configu¬ 
rations,  and  the  system  fielded  meets  the  user’s 
needs. 
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MODULE 


OPERATIONAL 
TEST  AND  EVALUATION 


Operational  Test  and  Evaluation  (OT&E)  is  conducted  to  ensure  a 
weapon  system  meets  the  validated  requirements  of  the  user  in  a 
realistic  scenario.  Operational  tests  are  focused  on  operational 
requirements,  effectiveness  and  suitability,  and  not  on  the  proof  of 
engineering  specifications,  as  is  the  case  with  development  testing. 
This  module  provides  an  overview  of  OT&E  and  discusses  how 
OT&E  results  provide  essential  information  for  milestone  decisions. 


INTRODUCTION  TO  OPERATIONAL 
TEST  AND  EVALUATION 


11.1  INTRODUCTION 

This  chapter  provides  an  introduction  to  the  con¬ 
cept  of  Operational  Test  and  Evaluation  (OT&E). 
It  outlines  the  purpose  of  OT&E,  discusses  the 
primary  participants  in  the  OT&E  process,  de¬ 
scribes  several  types  of  OT&E,  and  includes  some 
general  guidelines  for  the  successful  planning, 
execution,  and  reporting  of  OT&E  programs. 

11.2  PURPOSE  OF  OT&E 

OT&E  is  conducted  for  major  programs  by  an  or¬ 
ganization  that  is  independent  of  the  developing, 
procuring,  and  using  commands.  Some  form  of 
Operational  Assessment  (OA)  is  normally  con¬ 
ducted  in  each  acquisition  phase.  Each  assessment 
should  be  keyed  to  a  decision  review  in  the  mate¬ 
riel  acquisition  process.  It  should  include  typical 
user  operators,  crews,  or  units  in  realistic  combat 
simulations  of  operational  environments.  The  OT&E 
provides  the  decision  authority  with  an  estimate  of: 

(1)  The  degree  of  satisfaction  of  the  user’s  re¬ 
quirements  expressed  as  operational  effec¬ 
tiveness  and  operational  suitability  of  the 
new  system; 

(2)  The  system’s  desirability,  considering  equip¬ 
ment  already  available,  and  operational  ben¬ 
efits  or  burdens  associated  with  the  new 
system; 

(3)  The  need  for  further  development  of  the  new 
system  to  correct  performance  deficiencies; 

(4)  The  adequacy  of  doctrine,  organizations, 
operating  techniques,  tactics,  and  training  for 


employment  of  the  system;  of  maintenance 
support  for  the  system;  and  of  the  system’s 
performance  in  the  countermeasures 
environment. 

11.3  TEST  PARTICIPANTS 

The  OT&E  of  developing  systems  is  managed  by 
an  independent  Operational  Test  Agency  (OTA), 
which  each  Service  is  required  to  maintain.  It  is 
accomplished  under  conditions  of  operational  re¬ 
alism  whenever  possible.  Personnel  who  operate, 
maintain,  and  support  the  system  during  OT&E 
are  trained  to  a  level  commensurate  with  that  of 
personnel  who  will  perform  these  functions  un¬ 
der  peacetime  and  wartime  conditions.  Also,  Pro¬ 
gram  Management  Office  (PMO)  personnel,  the 
Integrated  Product  Teams  (IPTs),  and  test  coordi¬ 
nating  groups  play  important  parts  in  the  overall 
OT&E  planning  and  execution  process. 

11.3.1  Service  Operational  Test  Agencies 
(OTAs) 

The  OT&E  agencies  should  become  involved  early 
in  the  system’s  life  cycle,  usually  during  the 
program’s  evaluation  of  concepts.  At  this  time,  they 
can  begin  to  develop  strategies  for  conducting  op¬ 
erational  tests  (OT&E).  As  test  planning  contin¬ 
ues,  a  more-detailed  Test  and  Evaluation  Master 
Plan  (TEMP),  Part  IV  (OT&E),  is  developed  and 
the  test  resources  are  identified  and  scheduled. 
During  the  early  stages,  the  OTAs  structure  an 
OT&E  program  consistent  with  the  approved  ac¬ 
quisition  strategy  for  the  system,  identify  critical 
Operational  Test  (OT)  issues,  and  assess  the  ad¬ 
equacy  of  candidate  systems.  As  the  program 
moves  into  advanced  planning,  OT&E  efforts  are 


directed  toward  becoming  familiar  with  the  sys¬ 
tem,  encouraging  interface  between  the  user  and 
developer  and  further  refining  the  Critical  Opera¬ 
tional  Issues  (COIs).  The  OTA  test  directors,  ana¬ 
lysts,  and  evaluators  design  the  OT&E  so  that  the 
data  collected  will  support  answering  the  COIs. 
Each  Service  has  an  independent  organization 
dedicated  to  planning,  executing,  and  reporting  the 
results  of  that  Service’s  OT&E  activities.  These 
organizations  are  the:  Army  Test  and  Evaluation 
Command  (ATEC),  Navy  Operational  Test  and 
Evaluation  Force  (OPTEVFOR),  Air  Force  Op¬ 
erational  Test  and  Evaluation  Center  (AFOTEC), 
and  Marine  Corps  Operational  Test  and  Evalua¬ 
tion  Activity  (MCOTEA).  Although  policy  only 
requires  OTA  within  the  Services,  the  Defense 
Information  Systems  Agency  (DISA)  has  desig¬ 
nated  the  Joint  Interoperability  Test  Command 
(JITC)  as  the  OTA  for  their  DoD  information  sys¬ 
tems  and  a  Special  Operations  Command 
(SOCOM)  element  conducts  many  of  its  own 
IOT&E. 

11.3.2  Test  Personnel 

Operational  testing  is  conducted  on  materiel  sys¬ 
tems  with  “typical”  user  organizational  units  in  a 
realistic  operational  environment.  It  uses  person¬ 
nel  with  the  same  Military  Occupational  Special¬ 
ties  (MOSs)  as  those  who  will  operate,  maintain, 
and  support  the  system  when  fielded.  Participants 
are  trained  in  the  system’s  operation  based  on  the 
Service’s  operational  mission  profiles.  Because 
some  OT&E  consist  of  force-on-force  tests,  the 
forces  opposing  the  tested  system  must  also  be 
trained  in  the  use  of  threat  equipment,  tactics,  and 
doctrine.  For  operational  testing  conducted  before 
Initial  Operational  Test  and  Evaluation  (IOT&E), 
most  system  training  is  conducted  by  the  system’s 
contractor.  For  IOT&E,  the  contractor  trains  the 
Service  school  cadre  who  then  train  the  partici¬ 
pating  organizational  units.  Once  the  system  has 
entered  Full-Rate  Production  (FRP),  the  Service 
will  normally  assume  training  responsibilities. 
Operational  testing  often  requires  a  large  support 
staff  of  data  collectors  and  scenario  controllers 


operating  in  the  field  with  the  user  test  forces  and 
opposing  forces. 

11.4  TYPES  OF  OT&E 

The  OT&E  can  be  subdivided  into  two  phases: 
operational  testing  performed  before  FRP  and  the 
operational  testing  performed  after  the  FRP  deci¬ 
sion.  The  pre-FRP  OT&E  includes  operational  as¬ 
sessments  (EOAs,  OAs)  and  IOT&E.  OAs  begin 
early  in  the  program,  frequently  before  program 
start  and  continue  until  the  system  is  certified  as 
ready  for  IOT&E.  The  initial  IOT&E  is  conducted 
late  during  low  rate  production,  when  test  assets 
are  available,  in  support  of  the  full  rate  decision 
review.  The  Navy  uses  the  term  “OPEVAL”  (Op¬ 
erational  Evaluation)  to  define  IOT&E.  After  tran¬ 
sition  to  FRP,  all  subsequent  operational  testing  is 
usually  referred  to  as  Follow-on  Operational  Test 
and  Evaluation  (FOT&E).  In  the  Air  Force,  if  no 
Research  and  Development  (R&D)  funding  is 
committed  to  a  system,  Qualification  Operational 
Test  and  Evaluation  (QOT&E)  may  be  performed 
in  lieu  of  IOT&E. 

11.4.1  Early  Operational  Assessments 

The  EOAs  are  conducted  primarily  to  forecast  and 
evaluate  the  potential  operational  effectiveness  and 
suitability  of  the  weapon  system  during  develop¬ 
ment.  Early  OAs  start  during  Concept  Refinement 
(CR)  and/or  Technology  Development  (TD),  may 
continue  into  system  integration,  and  are  conducted 
on  prototypes  of  the  developing  design. 

11.4.1.1  Operational  Assessments  (OAs) 

The  OAs  begin  when  the  OTAs  start  their  evalu¬ 
ations  of  system-level  performance.  The  OTA  uses 
any  testing  results,  Modeling  and  Simulation 
(M&S),  and  data  from  other  sources  during  an 
assessment.  These  data  are  evaluated  by  the  OTA 
from  an  operational  point  of  view.  As  the  pro¬ 
gram  matures,  these  OAs  of  performance  require¬ 
ments  are  conducted  on  Engineering  Development 
Models  (EDMs)  or  pre-production  articles  until 
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the  system  performance  is  considered  mature.  The 
mature  system  can  then  be  certified  ready  for  its 
IOT&E  (OPEVAL  in  the  Navy).  When  using  an 
evolutionary  acquisition  strategy,  subsequent  in¬ 
crements  must  have  at  least  an  OA  before  the  new 
increment  is  issued  to  the  user. 

11.4.1.2  Initial  Operational  Test  and 
Evaluation  (Navy  OPEVAL) 

The  IOT&E  is  the  final  dedicated  phase  of  OT&E 
preceding  a  FRP  decision.  It  is  the  final  evalua¬ 
tion  that  entails  dedicated  operational  testing  of 
production-representative  test  articles  and  uses  typi¬ 
cal  operational  personnel  in  a  scenario  that  is  as 
realistic  as  possible  in  compliance  with  10  U.S.C. 
2399  ( Operational  Test  and  Evaluation  of  Defense 
Acquisition  Programs).  The  IOT&E  is  conducted 
by  an  OT&E  agency  independent  of  the  contrac¬ 
tor,  PMO,  or  developing  agency.  The  test  has  been 
described  as: 

All  OT&E  conducted  on  production  or  pro¬ 
duction  representative  articles,  to  support 
the  decision  to  proceed  beyond  Low  Rate 
Initial  Production  [LRIP].  It  is  conducted 
to  provide  a  valid  estimate  of  expected  sys¬ 
tem  operational  effectiveness  and  opera¬ 
tional  suitability.  The  definition  of 
“OT&E”  as  spelled  out  in  congressional 
legislation  (see  glossary)  is  generally  con¬ 
sidered  applicable  only  to  Initial  Opera¬ 
tional  Test  and  Evaluation  (IOT&E). 

Further,  IOT&E  must  be  conducted  with¬ 
out  system  contractor  personnel  participa¬ 
tion  in  any  capacity  other  than  stipulated 
in  service  wartime  tactics  and  doctrine  as 
set  forth  in  Public  Law  99-661  [National 
Defense  Authorization  Act  (NDAA),  Sec¬ 
tion  1207]  by  Congress.  The  results  from 
this  test  are  evaluated  and  presented  to  the 
Milestone  Decision  Authority  [MDA]  (i.e. 
the  decision  to  enter  FRP)  to  support  the 
Beyond-Low-Rate  Initial  Production 
(BLRIP)  decision.  This  phase  of  OT&E 


addresses  the  Key  Performance  Parameters 
[KPPs]  identified  in  the  Capability  Produc¬ 
tion  Document  (CPD)  and  the  Critical 
Operational  Issues  [COIs]  in  the  TEMP. 
IOT&E  test  plans  for  ACAT I  and  IA  and 
other  designated  programs  must  be  ap¬ 
proved  by  the  OSD  [Office  of  the  Secre¬ 
tary  of  Defense]  Director,  Operational 
Test  and  Evaluation  [DOT&E].  Service 
IOT&E  test  reports  provide  the  founda¬ 
tion  for  the  DOT&E  BLRIP  Report. 

11.4.2  Follow-On  Operational  Test  and 
Evaluation  (FOT&E) 

The  FOT&E  is  conducted  after  the  FRP  decision. 
The  tests  are  conducted  in  a  realistic  tactical  envi¬ 
ronment  similar  to  that  used  in  IOT&E,  but  fewer 
test  items  may  be  used.  Normally  FOT&E  is  con¬ 
ducted  using  fielded  production  systems  with  ap¬ 
propriate  modifications,  upgrades,  or  increments. 
Specific  objectives  of  FOT&E  include  testing 
modifications  that  are  to  be  incorporated  into  pro¬ 
duction  systems,  completing  any  deferred  or  in¬ 
complete  IOT&E,  evaluating  correction  of  defi¬ 
ciencies  found  during  IOT&E,  and  assessing  reli¬ 
ability  including  spares  support  on  deployed  sys¬ 
tems.  The  tests  are  also  used  to  evaluate  the  sys¬ 
tem  in  a  different  platform  application  for  new 
tactical  applications  or  against  new  threats. 

11.4.3  Qualification  Operational  Test  and 
Evaluation  (QOT&E)  (USAF) 

Air  Force  QOT&E  may  be  performed  by  the 
major  command,  user,  or  AFOTEC  and  is  con¬ 
ducted  on  minor  modifications  or  new  applica¬ 
tions  of  existing  equipment  when  no  Research  and 
Development  (R&D)  funding  is  required.  An  ex¬ 
ample  of  a  program  in  which  QOT&E  was  per¬ 
formed  by  the  Air  Force  is  the  A- 10  Air-to-Air 
Self  Defense  Program.  In  this  program  the  mis¬ 
sion  of  the  A- 10  was  expanded  from  strictly 
ground  support  to  include  an  air-to-air  defense 
role.  To  accomplish  this,  the  A- 10  aircraft  was 
modified  with  off-the-shelf  AIM-9  and  air-to-air 


11-3 


missiles;  QOT&E  was  performed  on  the  system 
to  evaluate  its  operational  effectiveness  and  suit¬ 
ability. 

11.5  TEST  PLANNING 

Operational  Test  (OT)  planning  is  one  of  the  most 
important  parts  of  the  OT&E  process.  Proper  plan¬ 
ning  facilitates  the  acquisition  of  data  to  support 
the  determination  of  the  weapon  system’s  opera¬ 
tional  effectiveness  and  suitability.  Planning  must 
be  pursued  in  a  deliberate,  comprehensive,  and 
structured  manner.  Careful  and  complete  planning 
may  not  guarantee  a  successful  test  program;  but 
inadequate  planning  can  result  in  significant  test 
problems,  poor  system  performance,  and  cost  over¬ 
runs.  OT  planning  is  conducted  by  the  OTA  be¬ 
fore  program  start,  and  more-detailed  planning 
usually  starts  about  two  years  before  each  OT 
event. 

Operational  planning  can  be  divided  into  three 
phases:  early  planning,  advanced  planning,  and 
detailed  planning.  Early  planning  entails  develop¬ 
ing  COIs,  formulating  a  plan  for  evaluations,  de¬ 
termining  the  Concept  of  Operation  (COO),  envi¬ 
sioning  the  operational  environment  and  develop¬ 
ing  mission  scenarios  and  resource  requirements. 
Advanced  planning  encompasses  the  determina¬ 
tion  of  the  purpose  and  scope  of  testing  and  iden¬ 
tification  of  Measures  of  Effectiveness  (MOEs) 
for  critical  issues.  It  includes  developing  test  ob¬ 
jectives,  establishing  a  test  approach,  and  estimat¬ 
ing  test  resource  requirements.  Detailed  planning 
involves  developing  step-by-step  procedures  to  be 
followed  as  well  as  the  final  coordination  of  re¬ 
source  requirements  necessary  to  carry  out  OT&E. 

11.5.1  Testing  Critical  Operational  Issues 
(COIs) 

The  COIs  have  been  described  as: 

A  key  operational  effectiveness  or  opera¬ 
tional  suitability  issue  that  must  be  exam¬ 
ined  in  OT&E  to  determine  the  system’s 


capability  to  perform  its  mission.  A  COI  is 
normally  phrased  as  a  question  to  be  an¬ 
swered  in  evaluating  a  system’s  operational 
effectiveness  and/or  operational  suitability. 

One  of  the  purposes  of  OT&E  is  to  resolve  COIs 
about  the  system.  The  first  step  in  an  OT&E  pro¬ 
gram  is  to  identify  these  critical  issues,  some  of 
which  are  explicit  in  the  Operational  Requirement 
Document  (ORD).  Examples  can  be  found  in 
questions  such  as:  “How  well  does  the  system  per¬ 
form  a  particular  aspect  of  its  mission?”  “Can  the 
system  be  supported  logistically  in  the  field?” 
Other  issues  arise  from  questions  asked  about  sys¬ 
tem  performance  or  how  it  will  affect  other  sys¬ 
tems  with  which  it  must  operate.  Critical  issues 
provide  focus  and  direction  for  the  OT.  Identify¬ 
ing  the  issues  is  analogous  to  the  first  step  in  the 
System  Engineering  Process  (SEP),  that  is,  defin¬ 
ing  the  problem.  When  COIs  are  properly  ad¬ 
dressed,  deficiencies  in  the  system  can  be  uncov¬ 
ered.  They  form  the  basis  for  a  structured  tech¬ 
nique  of  analysis  by  which  detailed  sub-objectives 
or  MOEs  can  be  established.  During  the  OT,  each 
sub-objective  is  addressed  by  an  actual  test  mea¬ 
surement  (Measure  of  Performance  (MOP)).  Af¬ 
ter  these  issues  are  identified,  the  evaluation  plans 
and  test  design  are  developed  for  test  execution. 
(For  more  information,  see  Chapter  13.) 

11.5.2  Test  Realism 

Test  realism  for  OT&E  will  vary  directly  with  the 
degree  of  system  maturity.  Efforts  early  in  the  ac¬ 
quisition  program  should  focus  on  active  involve¬ 
ment  of  users  and  operationally  oriented  environ¬ 
ments.  Fidelity  of  the  “combat  environment” 
should  peak  during  the  IOT&E  when  force-on- 
force  testing  of  the  production  representative  sys¬ 
tem  is  conducted.  The  degree  of  success  in  repli¬ 
cating  a  realistic  operational  environment  has  a 
direct  impact  on  the  credibility  of  the  IOT&E  test 
report.  Areas  of  primary  concern  for  the  test  plan¬ 
ner  can  be  derived  from  the  legislated  definition 
of  OT&E: 
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(1)  A  field  test  includes  all  of  the  elements  nor¬ 
mally  expected  to  be  encountered  in  the 
operational  arena,  such  as  appropriate  size 
and  type  of  maneuver  terrain,  environmen¬ 
tal  factors,  day/night  operations,  austere 
living  conditions,  etc. 

(2)  Realistic  combat  should  be  replicated  using 
appropriate  tactics  and  doctrine,  representa¬ 
tive  threat  forces  properly  trained  in  the  em¬ 
ployment  of  threat  equipment,  free  play  re¬ 
sponses  to  test  stimulus,  stress,  “dirty”  battle 
area  (fire,  smoke,  Nuclear,  Biological  and 
Chemical  (NBC);  Electronic  Countermea¬ 
sures  (ECM),  etc.),  wartime  tempo  to  op¬ 
erations,  real  time  casualty  assessment,  and 
forces  requiring  interoperability. 

(3)  Any  item  means  the  production  representa¬ 
tive  configuration  of  the  system  at  that  point 
in  time,  including  appropriate  logistics  tail. 

(4)  Typical  military  users  are  obtained  by  tak¬ 
ing  a  cross  section  of  adequately  trained  skill 
levels  and  ranks  of  the  intended  operational 
force.  Selection  of  “golden  crews”  or  the 
best  of  the  best  does  not  provide  test  data 
reflecting  the  successes  nor  problems  of  the 
“Murphy  and  gang”  of  typical  units. 

In  his  book,  Operational  Test  and  Evaluation:  A 
Systems  Engineering  Process ,  Roger  Stevens 
stated,  “In  order  to  achieve  realism  effectively  in 
an  OT&E  program,  a  concern  for  realism  must 
pervade  the  entire  test  program  from  the  very  be¬ 
ginning  of  test  planning  to  the  time  when  the  very 
last  test  iteration  is  run.”  Realism  is  a  significant 
issue  during  planning  and  execution  of  OT&E 
(Reference  113). 

11.5.3  Selection  of  a  Test  Concept 

An  important  step  in  the  development  of  an 
OT&E  program  is  to  develop  an  overall  test  pro¬ 
gram  concept.  Determinations  must  be  made  re¬ 
garding  when  OT&E  will  be  conducted  during 


systems  development,  what  testing  is  to  be  done 
on  production  equipment,  how  the  testing  will  be 
evolutionary,  and  what  testing  will  have  to  wait 
until  all  system  capabilities  are  developed.  This 
concept  can  best  be  developed  by  considering  a 
number  of  aspects  such  as  test  information  require¬ 
ments,  system  availability  for  test  periods,  and  the 
demonstration  of  system  capabilities.  The  test  con¬ 
cept  is  driven  by  the  acquisition  strategy  and  is  a 
road  map  used  for  planning  T&E  events.  The 
DOT&E  is  briefed  on  test  concepts  for  oversight 
programs  and  approves  the  test  plan  before 
IOT&E  starts. 

11.6  TEST  EXECUTION 

An  OT  plan  is  only  as  good  as  the  execution  of 
that  plan.  The  execution  is  the  essential  bridge 
between  test  planning  and  test  reporting.  The  test 
is  executed  through  the  OTA  test  director’s  ef¬ 
forts  and  the  actions  of  the  test  team.  For  suc¬ 
cessful  execution  of  the  OT&E  plan,  the  test  di¬ 
rector  must  direct  and  control  the  test  resources 
and  collect  the  data  required  for  presentation  to 
the  decision  authority.  The  test  director  must  pre¬ 
pare  for  testing,  activate,  and  train  the  test  team, 
develop  test  procedures  and  operating  instruc¬ 
tions,  control  data  management,  create  OT&E 
plan  revisions,  and  manage  each  of  the  test  tri¬ 
als.  The  test  director’s  data  management  duties 
will  encompass  collecting  raw  data,  creating  a 
data  status  matrix,  and  ensuring  data  quality  by 
processing  and  reducing,  verifying,  filing,  stor¬ 
ing,  retrieving,  and  analyzing  collected  data. 
Once  all  tests  have  been  completed  and  the  data 
is  reduced  and  analyzed,  the  results  must  be  re¬ 
ported.  A  sample  test  organization  used  for  the 
Army  OT&E  of  the  program  illustrated  in  Fig¬ 
ure  11-1.  (In  the  Army,  the  Deputy  Test  Direc¬ 
tor  comes  from  the  OTA  and  controls  the  daily 
OT  activity.) 

11.7  TEST  REPORTING 

The  IOT&E  test  report  is  a  very  important  docu¬ 
ment.  It  must  communicate  the  results  of  completed 
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Figure  11-1.  Organizational  Breakdown  of  the  1-81  mm 
Mortar  Operational  Test  Directorate 


tests  to  decision  authorities  in  a  timely,  factual, 
concise,  comprehensive,  and  accurate  manner.  The 
report  must  present  a  balanced  view  of  the  weapon 
system’s  successes  and  problems  during  testing, 
illuminating  both  the  positive  aspects  and  system 
deficiencies  discovered.  Analysis  of  test  data  and 
their  evaluation  may  be  in  one  report  (USAF, 
USN)  or  in  separate  documents  (USA,  USMC). 
The  Army’s  Independent  Evaluation  Report  (IER) 
advises  the  decision  review  principals  and  MDA 
concerning  the  adequacy  of  testing,  the  system’s 
operational  effectiveness  and  suitability,  and  sur¬ 
vivability,  as  well  as  providing  recommendations 
for  future  T&E  and  system  improvements. 

There  are  four  types  of  reports  most  frequently 
used  in  reporting  OT&E  results.  These  include 


status,  interim,  quick-look,  and  final  reports.  The 
status  report  gives  periodic  updates  (e.g.,  monthly, 
quarterly)  and  reports  recent  test  findings  (discreet 
events  such  as  missile  firings).  The  interim  report 
provides  a  summary  of  the  cumulative  test  results 
to  date  when  there  is  an  extended  period  of  test¬ 
ing.  The  quick-look  reports  provide  preliminary 
test  results,  are  usually  prepared  immediately  af¬ 
ter  a  test  event  (less  than  seven  days)  and  have 
been  used  to  support  program  decision  milestones. 
The  final  T&E  report  (Air  Force,  Navy)  or  IER 
(Army,  Marine)  presents  the  conclusions  and  rec¬ 
ommendations  including  all  supporting  data  and 
covering  the  entire  IOT&E  program.  DoDI 
5000.2,  Operation  of  the  Defense  Acquisition  Sys¬ 
tem,  12  May  2003,  requires  the  Service  OTA 
provide  reports  of  OT&E  results  in  support  of 
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Milestones  B  and  C,  in  addition  to  the  IOT&E 
report  required  for  the  Full  Rate  Production  Deci¬ 
sion  Review  (FRPDR). 

11.8  SUMMARY 

The  purpose  of  OT&E  is  to  assess  operational 
effectiveness  and  suitability  at  each  stage  in  the 
acquisition  process.  Operational  effectiveness  is  a 
measure  of  the  contribution  of  the  system  to  mis¬ 
sion  accomplishment  under  actual  conditions  of 
employment.  Operational  suitability  is  a  measure 
of  the  Reliability  and  Maintainability  (R&M)  of 


the  system;  the  effort  and  level  of  training  required 
to  maintain,  support,  and  operate  it;  and  any 
unique  logistic  or  training  requirements  it  may 
have.  The  OT&E  may  provide  information  on  tac¬ 
tics,  doctrine,  organization,  and  personnel  require¬ 
ments  and  may  be  used  to  assist  in  the  prepara¬ 
tion  of  operating  and  maintenance  instructions  and 
other  publications.  One  of  the  most  important  as¬ 
pects  is  that  OT&E  provides  an  independent  evalu¬ 
ation  of  the  degree  of  progress  made  toward  sat¬ 
isfying  the  user’s  requirements  during  the  system 
development  process. 


11-7 


OT&E  TO  SUPPORT 
DECISION  REVIEWS 


12.1  INTRODUCTION 

Mindful  of  principles  of  objectivity  and  impartial 
evaluation,  Operational  Test  and  Evaluation  (OT&E) 
may  be  conducted  before  each  decision  review  to 
provide  the  decision  authority  with  the  latest  results 
from  testing  of  Critical  Operational  Issues  (COIs). 
The  philosophy  of  OT&E  has  been  related  to  three 
terms — adequacy,  quality,  and  credibility: 

Adequacy  —  The  amount  of  data  and  re¬ 
alism  of  test  conditions  must  be  sufficient 
to  support  the  evaluation  of  the  COIs. 

Quality  —  The  test  planning,  control  of 
test  events,  and  treatment  of  data  must  pro¬ 
vide  clear  and  accurate  test  reports. 

Credibility  —  The  conduct  of  the  test  and 
data  handling  must  be  separated  from  ex¬ 
ternal  influence  and  personal  biases. 

Operational  testing  is  conducted  to  provide  infor¬ 
mation  to  support  Department  of  Defense  (DoD) 
executive-level  management  decisions  on  major 
acquisition  programs.  OT&E  is  accomplished  us¬ 
ing  a  test  cycle  of  successive  actions  and  documents. 
During  the  early  stages  of  the  program,  the  process 
is  informal  and  modified  as  necessary.  As  programs 
mature,  documentation  for  major  systems  and  those 
designated  by  the  Director,  Operational  Test  and 
Evaluation  (DOT&E)  for  oversight  must  be  sent  to 
the  Office  of  the  Secretary  of  Defense  (OSD)  for 
approval  before  the  testing  can  be  conducted  or 
the  systems  can  be  cleared  to  proceed  Beyond  Low 
Rate  Initial  Production  (BLRIP).  Figure  12-1 
illustrates  how  OT&E  relates  to  the  acquisition 
process. 


12.2  CONCEPT  REFINEMENT  (CR)  AND 
TECHNOLOGY  DEVELOPMENT  (TD) 

The  OT&E  conducted  during  the  first  phase  may 
be  an  Early  Operational  Assessment  (EOA)  fo¬ 
cused  on  investigating  the  deficiencies  identified 
during  the  mission  area  analysis.  Operational  testers 
participate  in  these  evaluations  to  validate  the 
OT&E  requirements  for  future  testing  and  to  iden¬ 
tify  issues  and  criteria  that  can  only  be  resolved 
through  OT&E  to  initiate  early  test  resource 
planning. 

Before  program  initiation,  the  OT&E  objectives 
are  to  assist  in  evaluating  alternative  concepts  to 
solve  the  mission  capability  deficiencies  and  to 
assess  the  operational  impact  of  the  system.  An 
early  assessment  also  may  provide  data  to  sup¬ 
port  a  decision  on  whether  the  technology  is  suf¬ 
ficiently  mature  to  support  entry  into  the  next  de¬ 
velopment  phase.  The  OT&E  conducted  during 
this  phase  supports  developing  estimates  of: 

(1)  The  military  need  for  the  proposed  system; 

(2)  A  demonstration  that  there  is  a  sound  physi¬ 
cal  basis  for  a  new  system; 

(3)  An  analysis  of  concepts,  based  on  demon¬ 
strated  physical  phenomena,  for  satisfying 
the  military  need; 

(4)  The  system’s  affordability  and  Life  Cycle 
Cost  (LCC); 

(5)  The  ability  of  a  modification  to  an  existing  U.S. 
or  allied  system  to  provide  needed  capability; 
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Figure  12-1.  OT&E  Related  to  the  Milestone  Process 


(6)  An  operational  utility  assessment; 

(7)  An  impact  of  the  system  on  the  force 
structure. 

During  concept  assessment,  there  is  normally  no 
hardware  available  for  the  operational  tester.  There¬ 
fore,  the  EOA  is  conducted  from  surrogate  test 
and  experiment  data,  breadboard  models,  factory 
user  trials,  mock-up/simulators,  modeling/simula- 
tion,  and  user  demonstrations  (Figure  12-2).  This 
makes  early  assessments  difficult,  and  some  areas 
cannot  be  covered  in-depth.  However,  these  as¬ 
sessments  provide  vital  introductory  information 
on  the  system’s  potential  operational  utility. 

The  OT&E  products  from  this  phase  of  testing 
include  the  information  provided  to  the  decision 
authority,  data  collected  for  further  evaluation,  in¬ 
put  to  the  test  planning  in  the  Technology  Devel¬ 
opment  Strategy  (TDS)  that  will  later  evolve  into 
the  Test  and  Evaluation  Master  Plan  (TEMP)  and 
early  Test  and  Evaluation  (T&E)  planning.  Spe¬ 
cial  logistics  problems,  program  objectives,  pro¬ 
gram  plans,  performance  parameters,  and  acquisi¬ 
tion  strategy  are  areas  of  primary  influence  to  the 
operational  tester  during  this  phase  and  must  be 


carefully  evaluated  to  project  the  system’s 
operational  effectiveness  and  suitability. 

12.3  SYSTEM  DEVELOPMENT  AND 
DEMONSTRATION  (SDD) 

After  program  initiation,  combined  DT/OT&E  or  an 
EOA  may  be  conducted  to  support  the  EOA  of  pro¬ 
totype  development  and  a  decision  regarding  a 
system’s  readiness  to  move  into  the  development  of 
the  Engineering  Development  Model  (EDM).  In  all 
cases,  appropriate  T&E  must  be  conducted  on  a 
mature  system  configuration,  i.e.  the  EDM,  before 
the  production  decision,  thereby  providing  data  for 
identification  of  risk  before  more  resources  are  com¬ 
mitted.  As  appropriate,  Low  Rate  Initial  Production 
(LRIP)  will  follow  this  phase  of  testing  to  verify 
system  level  performance  capability  and  to  provide 
insight  into  test  resources  needed  to  conduct  future 
interoperability,  live  fire,  or  operational  testing. 

12.3.1  Objectives  of  Operational  Assessments 

Operational  Assessments  (EOAs,  OAs)  are 
conducted  to  facilitate  identification  of  the  best  design, 
indicate  the  risk  level  of  performance  for  this  phase 
of  the  development,  examine  operational  aspects 
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of  the  system’s  development,  and  estimate  poten¬ 
tial  operational  effectiveness  and  suitability.  Addi¬ 
tionally,  an  analysis  of  the  planning  for  transition 
from  development  to  production  is  initiated.  EOAs 
supporting  decision  reviews  are  intended  to: 

(1)  Assess  the  potential  of  the  new  system  in 
relation  to  existing  capabilities; 

(2)  Assess  system  effectiveness  and  suitability 
so  that  affordability  can  be  evaluated  for 
program  cost  versus  military  utility; 

(3)  Assess  the  adequacy  of  the  concept  for  em¬ 
ployment,  supportability,  and  organization; 
doctrinal,  tactical,  and  training  requirements; 
and  related  critical  issues; 

(4)  Estimate  the  need  for  the  selected  systems 
in  consideration  of  the  threat  and  system  al¬ 
ternatives  based  on  military  utility; 

(5)  Assess  the  validity  of  the  operational  concept; 

(6)  List  the  key  risk  areas  and  COIs  that  need 
to  be  resolved  before  construction  of  EDMs 
is  initiated; 

(7)  Assess  the  need  during  LRIP  of  long  lead 
hardware  to  support  Initial  Operational  Test 


and  Evaluation  (IOT&E)  prior  to  the  Full- 
Rate  Production  (FRP)  decision; 

(8)  Provide  data  to  support  test  planning  for  this 
phase. 

During  this  phase,  OT&E  may  be  conducted  on 
brassboard  configurations,  experimental  proto¬ 
types  or  advanced  development  prototypes.  Dedi¬ 
cated  test  time  may  be  made  available  for  the 
operational  tester.  However,  the  OT&E  assess¬ 
ments  may  also  make  use  of  many  other  addi¬ 
tional  data  sources.  Examples  of  additional 
sources  often  used  by  the  Army  during  this  phase 
include:  Concept  Evaluation  Program  (CEP) 
tests,  innovative  testing,  Force  Development 
Tests  and  Experimentation  (FDT&Es),  source 
selection  tests,  user  participation  in  Development 
Test  and  Evaluation  (DT&E)  and  operational 
feasibility  tests.  The  results  from  this  testing, 
analysis,  and  evaluation  are  documented  in  an 
Operational  Assessment  (EOA)  or  end  of  phase 
OT&E  report.  These  data,  along  with  the  mis¬ 
sion  needs  and  requirements  documentation  and 
TEMP,  assist  in  the  review  of  performance  for 
the  next  decision  review. 

The  OAs  during  the  system  demonstration  are  con¬ 
ducted  on  EDMs.  These  operational  evaluations 
estimate  the  operational  effectiveness  and  suitability 
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and  provide  data  on  whether  the  system  meets 
minimum  operational  thresholds 

12.4  PRODUCTION  AND  DEPLOYMENT 
(P&D) 

Just  before  the  FRP  decision,  the  dedicated  T&E 
is  conducted  on  equipment  that  has  been  formally 
certified  by  the  Program  Manager  (PM)  as  being 
ready  for  the  “final  OT&E.”  This  dedicated 
IOT&E  is  conducted  in  a  test  environment  as  op¬ 
erationally  realistic  as  possible. 

12.4.1  OT&E  Objectives 

The  IOT&E  conducted  is  characterized  by  testing 
performed  by  user  organizations  in  a  field  exercise 
to  examine  the  organization  and  doctrine,  Integrated 
Logistics  Support  (ILS),  threat,  communications, 
Command  and  Control  (C2),  and  tactics  associated 
with  the  operational  employment  of  the  unit  during 
tactical  operations.  This  includes  estimates  which: 

(1)  Assess  operational  effectiveness  and 
suitability; 

(2)  Assess  the  survivability  of  the  system; 

(3)  Assess  the  systems  reliability,  maintainability, 
and  plans  for  ILS; 

(4)  Evaluate  manpower,  personnel,  training,  and 
safety  requirements; 

(5)  Validate  organizational  and  employment 
concepts; 

(6)  Determine  training  and  logistics  requirements 
deficiencies; 

(7)  Assess  the  system’s  readiness  to  enter  FRP. 

12.5  OPERATIONS  AND  SUPPORT  (O&S) 

After  the  FRP  decision  and  deployment,  the 
emphasis  shifts  towards  procuring  production 


quantities,  repairing  hardware  deficiencies,  man¬ 
aging  changes,  and  phasing  in  full  logistics  sup¬ 
port.  During  initial  deployment  of  the  system,  the 
OT&E  agency  and/or  the  user  may  perform  Fol- 
low-on  Operational  Test  and  Evaluation  (FOT&E) 
to  refine  the  effectiveness  and  suitability  estimates 
made  during  earlier  OT&E,  assess  performance 
not  evaluated  during  IOT&E,  evaluate  new  tac¬ 
tics  and  doctrine,  and  assess  the  impacts  of  sys¬ 
tem  modifications  or  upgrades. 

The  FOT&E  is  performed  with  production  articles 
in  operational  organizations.  It  is  normally  funded 
with  Operations  and  Maintenance  (O&M)  funds. 
The  first  FOT&E  conducted  during  this  phase  may 
be  used  to: 

(1)  Ensure  that  the  production  system  performs 
as  well  as  reported  at  the  Milestone  III 
review; 

(2  Demonstrate  expected  performance  and 
reliability  improvements; 

(3)  Ensure  that  the  correction  of  deficiencies 
identified  during  earlier  testing  have  been 
completed; 

(4)  Evaluate  performance  not  tested  during 
IOT&E. 

Additional  objectives  of  FOT&E  are  to  validate 
the  operational  effectiveness  and  suitability  of  a 
modified  system  during  an  OA  of  the  system  in 
new  environments.  The  FOT&E  may  look  at  dif¬ 
ferent  platform  applications,  new  tactical  applica¬ 
tions,  or  the  impact  of  new  threats. 

12.5.1  FOT&E  of  Logistic  Support  Systems 

The  testing  objectives  to  evaluate  post-production 
logistics  readiness  and  support  are  to: 

(1)  Assess  the  logistics  readiness  and 
sustainability; 
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(2)  Evaluate  the  weapon  support  objectives; 


12.6  SUMMARY 


(3)  Assess  the  implementation  of  logistics 
support  planning; 

(4)  Evaluate  the  capability  of  the  logistics 
support  activities; 

(5)  Determine  the  disposition  of  displaced 
equipment; 

(6)  Evaluate  the  affordability  and  LCC  of  the 
system. 


The  OT&E  is  that  T&E  (OAs,  IOT&E,  or  FOT&E) 
conducted  to  provide  feedback  on  system  design 
on  potential  to  be  operational  effective  and  opera¬ 
tional  suitable.  OT&E  events  in  each  phase  of 
development  will  identify  needed  modifications; 
provide  information  on  tactics,  doctrine,  organi¬ 
zations,  and  personnel  requirements;  and  evaluate 
the  system’s  logistic  supportability.  The  acquisi¬ 
tion  program  structure  should  include  feedback 
from  OAs  or  evaluations  at  critical  decision  points/ 
technical  reviews  beginning  early  in  the  develop¬ 
ment  cycle  and  continuing  throughout  the  system’s 
life  cycle. 
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Many  program  managers  face  several  T&E  issues  that  must  be 
resolved  to  get  their  particular  weapon  system  tested  and  ultimately 
fielded.  These  issues  may  include  modeling  and  simulation  support, 
combined  and  concurrent  testing,  test  resources,  survivability  and 
lethality  testing,  multi-Service  testing,  or  international  T&E.  Each 
issue  presents  a  unique  set  of  challenges  for  the  program  manager 
when  he/she  develops  the  integrated  strategy  for  the  T&E  program. 


EVALUATION 


13.1  INTRODUCTION 

This  chapter  describes  the  evaluation  portion  of 
the  Test  and  Evaluation  (T&E)  process.  It  stresses 
the  importance  of  establishing  and  maintaining  a 
clear  audit  trail  from  system  requirements  through 
critical  issues,  evaluation  criteria,  test  objectives, 
and  Measures  of  Effectiveness  (MOEs)  to  the 
evaluation.  The  importance  of  the  use  of  data  from 
all  sources  is  discussed  as  are  the  differences  in 
approaches  to  evaluating  technical  and  operational 
data. 

13.2  DIFFERENCE  BETWEEN  “TEST” 
AND  “EVALUATION” 

The  following  distinction  has  been  made  between 
the  functions  of  “test”  and  “evaluation:” 

While  the  terms  “test”  and  “evaluation”  are 
most  often  found  together,  they  actually 
denote  clearly  distinguishable  functions  in 
the  RDT&E  [Research,  Development, 
Test,  and  Evaluation]  process. 

“Test”  denotes  “Any  program  or  procedure  which 
is  designed  to  obtain,  verify,  or  provide  data  for 
the  evaluation  of  any  of  the  following:  1)  progress 
in  accomplishing  developmental  objectives;  2)  the 
performance,  operational  capability,  and  suitability 
of  systems,  subsystems,  components,  and  equip¬ 
ment  items;  and  3)  the  vulnerability  and  lethality  of 
systems,  subsystems,  components,  and  equipment 
items.”  ( Glossary  of  Defense  Acquisition  Acronyms 
&  Terms,  1 1th  Edition,  September  2003.) 

“Evaluation”  denotes  the  process  whereby  data 
are  logically  assembled,  analyzed,  and  compared 


to  expected  performance  to  aid  in  making 
systematic  decisions. 

To  summarize,  evaluation  is  the  process  for  re¬ 
view  and  analysis  of  qualitative  or  quantitative  data 
obtained  from  design  review,  hardware  inspection, 
Modeling  and  Simulation  (M&S),  testing,  or 
operational  usage  of  equipment. 

13.3  THE  EVALUATION  PROCESS 

The  evaluation  process  requires  a  broad  analyti¬ 
cal  approach  with  careful  focus  on  the  develop¬ 
ment  of  an  overall  (T&E  plan  that  will  provide 
timely  answers  to  critical  issues  and  questions  re¬ 
quired  by  decision  authorities  throughout  all  the 
acquisition  phases.  (Table  13-1)  Evaluations 
should  focus  on  Key  Performance  Parameters 
(KPPs);  i.e.,  “those  minimum  attributes  or  char¬ 
acteristics  considered  most  essential  for  an  effec¬ 
tive  military  capability.”  An  attribute  is  a  testable 
or  measurable  characteristic  that  describes  an  as¬ 
pect  of  a  system  or  capability.  (Chairman  of  the 
Joint  Chiefs  of  Staff  Instruction  (CJCSI) 
3 170.01D,  Joint  Capabilities  Integration  and  De¬ 
velopment  System  (JCIDS),  12  March  2004.) 

A  functional  block  diagram  of  a  generic  (i.e.,  not 
Service-specific)  evaluation  process  is  shown  in 
Figure  13-1.  The  Army’s  Independent  Evaluation 
Plan  (IEP)  is  written  very  early  and  applies  to  the 
system  and  not  a  specific  test  event,  so  it  is  not  as 
detailed  as  the  format  provided  in  Table  13-1.  The 
process  begins  with  the  identification  of  a  defi¬ 
ciency  or  need  and  the  documentation  of  an  op¬ 
erational  requirement.  It  continues  with  the  iden¬ 
tification  of  critical  issues  that  must  be  addressed 
to  determine  the  degree  to  which  the  system  meets 
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Figure  13-1.  Functional  Block  Diagram  of  the  Evaluation  Process 


user  requirements.  Objectives  and  thresholds  must 
then  be  established  to  define  required  performance 
or  supportability  parameters  and  to  evaluate 
progress  in  reaching  them.  T&E  analysts  then 
decompose  the  issues  into  measurable  test  ele¬ 
ments,  conduct  the  necessary  testing,  review  and 
analyze  the  test  data,  weigh  the  test  results  against 
the  evaluation  criteria,  and  prepare  an  evaluation 
report  for  the  decision  authorities. 


(2)  Scope  —  detailed  conditions  and  range  of 
conditions  that  will  guide  the  T&E  process 
for  this  issue; 

(3)  Criteria —  quantitative  or  qualitative  stan¬ 
dards  that  will  answer  the  issue; 

(4)  Rationale  —  full  justification  to  support  the 
selected  criteria. 


13.4  ISSUES  AND  CRITERIA 

Issues  are  questions  regarding  a  system  that  re¬ 
quire  answers  during  the  acquisition  process. 
Those  answers  may  be  needed  to  aid  in  the  de¬ 
velopment  of  an  acquisition  strategy,  to  refine  per¬ 
formance  requirements  and  designs  or  to  support 
Milestone  Decision  Reviews  (MDRs).  Evaluation 
criteria  are  the  standards  by  which  accomplish¬ 
ments  of  required  technical  and  operational  effec¬ 
tiveness  and/or  suitability  characteristics  or  reso¬ 
lution  of  operational  issues  may  be  assessed.  The 
evaluation  program  may  be  constructed  using  a 
structured  approach  identifying  each  issue. 

(1)  Issue  —  a  statement  of  the  question  to  be 
answered; 


13.4.1  Key  Performance  Parameters  (KPPs)/ 
Critical  Issues 

The  KPPs  often  can  support  the  development  of 
a  hierarchy  of  critical  issues  and  less  significant 
issues.  Critical  issues  are  those  questions  relating 
to  a  system’s  operational,  technical,  support,  or 
other  capability.  These  issues  must  be  answered 
before  the  system’s  overall  worth  can  be  estimated/ 
evaluated,  and  they  are  of  primary  importance  to 
the  decision  authority  in  allowing  the  system  to 
advance  to  the  next  acquisition  phase.  Critical  is¬ 
sues  in  the  Test  and  Evaluation  Master  Plan 
(TEMP)  may  be  derived  from  the  KPPs  found  in 
the  capability  requirements  documents  (Capabil¬ 
ity  Development  Document  (CDD),  Capability 
Production  Document  (CPD)).  The  system 
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requirements  and  baseline  documentation  will  pro¬ 
vide  many  of  the  performance  parameters  required 
to  develop  the  hierarchy  of  issues. 

13.4.2  Evaluation  Issues 

Evaluation  issues  are  those  addressed  in  technical 
or  operational  evaluations  during  the  acquisition 
process.  Evaluation  issues  can  be  separated  into 
technical  or  operational  issues  and  addressed  in 
the  TEMP. 

Technical  issues  primarily  concern  technical  pa¬ 
rameters  or  characteristics  and  engineering  speci¬ 
fications  normally  assessed  in  development  test¬ 
ing.  Operational  issues  concern  effectiveness  and 
suitability  characteristics  for  functions  to  be  per¬ 
formed  by  equipment/personnel.  They  address  the 
system’s  operational  performance  when  examined 
in  a  realistic  operational  mission  environment. 
Evaluation  issues  are  answered  by  whatever 
means  necessary  (analysis/survey,  modeling,  simu¬ 
lation,  inspection,  demonstration,  or  testing)  to 
resolve  the  issue.  Issues  requiring  test  data  are 
further  referred  to  as  test  issues. 

13.4.3  Test  Issues 

Test  issues  are  a  subset  of  evaluation  issues  and 
address  areas  of  uncertainty  that  require  test  data 
to  resolve  the  issue  adequately.  Test  issues  may 
be  partitioned  into  technical  issues  that  are  ad¬ 
dressed  by  the  Development  Test  and  Evaluation 
(DT&E)  community  (contractor  and  government) 
and  operational  issues  that  are  addressed  by  the 
Operational  Test  and  Evaluation  (OT&E)  commu¬ 
nity.  Test  issues  may  be  divided  into  critical  and 
noncritical  categories.  All  critical  T&E  issues, 
objectives,  methodologies,  and  evaluation  criteria 
should  be  defined  during  the  initial  establishment 
of  an  acquisition  program.  Critical  issues  are  docu¬ 
mented  in  the  TEMP.  These  evaluation  issues  serve 
to  define  the  testing  required  for  each  phase  of 
the  acquisition  process  and  serve  as  the  structure 
to  guide  the  testing  program  so  these  data  may  be 
compared  against  performance  criteria. 


13.4.4  Criteria 

Criteria  are  statements  of  a  system’s  required  tech¬ 
nical  performance  and  operational  effectiveness, 
suitability,  and  supportability.  Criteria  are  often 
expressed  as  “objectives  and  thresholds.”  (Some 
Services,  however,  specify  performance  and  sup- 
portability  requirements  exclusively  in  terms  of 
thresholds  and  avoid  the  use  of  the  concept  of 
objectives.)  These  performance  measurements  pro¬ 
vide  the  basis  for  collecting  data  used  to  evaluate/ 
answer  test  issues. 

Criteria  must  be  unambiguous  and  assessable 
whether  stated  qualitatively  or  quantitatively.  They 
may  compare  the  mission  performance  of  the  new 
system  to  the  one  being  replaced,  compare  the  new 
system  to  a  predetermined  standard,  or  compare 
mission  performance  results  using  the  new  sys¬ 
tem  to  not  having  the  system.  Criteria  are  the  fi¬ 
nal  values  deemed  necessary  by  the  user.  As  such, 
they  should  be  developed  in  close  coordination 
with  the  system  user,  other  testers,  and  specialists 
in  all  other  areas  of  operational  effectiveness  and 
suitability.  These  values  may  be  changed  as  sys¬ 
tems  develop  and  associated  testing  and  evalua¬ 
tion  proceed.  Every  issue  should  have  at  least  one 
criteria  that  is  a  concise  measure  of  the  function. 
Values  must  be  realistic  and  achievable  within  the 
state  of  the  art  of  engineering  technology.  A  quan¬ 
titative  or  qualitative  criterion  should  have  a  clear 
definition,  free  of  ambiguous  or  imprecise  termi¬ 
nology,  such  as  “adequate,”  “sufficient,”  or 
“acceptable.” 

13.4.4.1  Test  of  Thresholds  and  Objectives 

A  threshold  for  a  performance  parameter  lists  a 
minimum  acceptable  operational  value  below 
which  the  utility  of  the  system  becomes  question¬ 
able.  (CJCSI  3 170.0 ID)  Thresholds  are  stated 
quantitatively  whenever  possible.  Specification  of 
minimally  acceptable  performance  in  measurable 
parameters  is  essential  to  selecting  appropriate 
MOEs,  which,  in  turn,  heavily  influence  test  de¬ 
sign.  Thresholds  are  of  value  only  when  they  are 
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testable;  i.e.,  actual  performance  can  be  measured 
against  them.  The  function  of  T&E  is  to  verify 
the  attainment  of  required  thresholds.  The  Program 
Manager  (PM)  must  budget  for  the  attainment  of 
threshold  values. 

Objectives  are  “the  desired  operational  goal  asso¬ 
ciated  with  a  performance  attribute,  beyond  which 
any  gain  in  utility  does  not  warrant  additional  ex¬ 
penditure.  The  objective  value  is  an  operationally 
significant  increment  above  the  threshold.  An  ob¬ 
jective  value  may  be  the  same  as  the  threshold 
when  an  operationally  significant  increment  above 
the  threshold  is  not  significant  or  useful.”  (CJCSI 
3170.01D)  Objectives  are  not  normally  addressed 
by  the  operational  tester,  whose  primary  concern 
is  to  “determine  if  thresholds  in  the  approved  CPD 
and  critical  operational  issues  have  been  satisfied.” 
(DoDI  5000.2,  Operation  of  the  Defense  Acqui¬ 
sition  System,  12  May  2003.) 

Going  into  system  demonstration,  thresholds  and 
objectives  are  expanded  along  with  the  identifica¬ 
tion  of  more-detailed  and  refined  performance 
capabilities  and  characteristics  resulting  from 
trade-off  studies  and  testing  conducted  during  the 
evaluation  of  Engineering  Development  Models 
(EDMs).  Along  with  the  CPD,  they  should  re¬ 
main  relatively  stable  through  production. 

13.5  MEASURES  OF  EFFECTIVENESS 
(MOEs) 

Requirements,  thresholds,  and  objectives  established 
in  early  program  documentation  form  the  basis  for 
evaluation  criteria.  If  program  documentation  is  in¬ 
complete,  the  tester  may  have  to  develop  evalua¬ 
tion  criteria  in  the  absence  of  specific  requirements. 
Evaluation  criteria  are  associated  with  objectives, 
sub-objectives  and  MOEs  (Sometimes  partitioned 
into  MOEs  and  measures  of  suitability).  For  ex¬ 
ample,  an  MOE  (e.g.,  airspeed)  may  have  an  asso¬ 
ciated  evaluation  criterion  (e.g.,  450  knots)  against 
which  the  actual  performance  (e.g.,  425  knots)  is 
compared  to  arrive  at  a  rating.  An  MOE  of  a  sys¬ 
tem  is  a  parameter  that  evaluates  the  capacity  of 


the  system  to  accomplish  its  assigned  missions 
under  a  given  set  of  conditions.  They  are  impor¬ 
tant  because  they  determine  how  test  results  will 
be  judged;  and,  since  test  planning  is  directed  to¬ 
ward  obtaining  these  measures,  it  is  important  that 
they  be  defined  early.  Generally,  the  resolution  of 
each  critical  issue  is  in  terms  of  the  evaluation  of 
some  MOE.  In  this  case,  the  operating,  implement¬ 
ing,  and  supporting  commands  must  agree  with 
the  criteria  before  the  test  organization  makes  use 
of  them  in  assessing  test  results.  Ensuring  that 
MOEs  can  be  related  to  the  user’s  operational  re¬ 
quirements  is  an  important  consideration  when 
identifying  and  establishing  evaluation  criteria. 
Testers  must  ensure  that  evaluation  criteria  and 
MOEs  are  updated  if  requirements  change.  The 
MOEs  should  be  so  specific  that  the  system’s  ef¬ 
fectiveness  during  developmental  and  operational 
testing  can  be  assessed  using  some  of  the  same 
effectiveness  criteria  as  the  Analysis  of  Alterna¬ 
tives  [AoAs].  (DoDI  5000.2) 

13.6  EVALUATION  PLANNING 

13.6.1  Evaluation  Planning  Techniques 

Evaluation  planning  is  an  iterative  process  that 
requires  formal  and  informal  analyses  of  system 
operation  (e.g.,  threat  environment,  system  design, 
tactics,  and  interoperability).  Techniques  that  have 
been  proven  effective  in  evaluation  planning  in¬ 
clude:  process  analysis,  design  or  engineering 
analysis,  matrix  analysis,  and  dendritic  analysis 
(Reference  60). 

13.6.1.1  Process  Analysis  Techniques 

Process  analysis  techniques  consist  of  thinking 
through  how  the  system  will  be  used  in  a  variety 
of  environments,  threats,  missions,  and  scenarios 
in  order  to  understand  the  events,  actions,  situa¬ 
tions,  and  results  that  are  expected  to  occur.  This 
technique  aids  in  the  identification  and  clarifica¬ 
tion  of  appropriate  MOEs,  test  conditions,  and 
data  requirements. 
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13.6.1.2  Design/Engineering  Analysis 
Techniques 

Design  or  engineering  analysis  techniques  are  used 
to  examine  all  mechanical  or  functional  operations 
that  the  system  has  been  designed  to  perform. 
These  techniques  involve  a  systematic  exploration 
of  the  system’s  hardware  and  software  compo¬ 
nents,  purpose,  performance  bounds,  manpower 
and  personnel  considerations,  known  problem  ar¬ 
eas,  and  impact  on  other  components.  Explora¬ 
tion  of  the  way  a  system  operates  compared  to 
intended  performance  functions  often  identifies  is¬ 
sues,  MOEs,  specific  data,  test  events,  and  required 
instrumentation. 

13.6.1.3  Matrix  Analysis  Techniques 

Matrix  analysis  techniques  are  useful  for  analyz¬ 
ing  any  situation  where  two  classifications  must 
be  cross-referenced.  For  example,  a  matrix  of 
“types  of  data”  versus”  means  of  data  collection” 
can  reveal  not  only  types  of  data  having  no  planned 
means  of  collection  but  also  redundant  or  backup 
collection  systems.  Matrix  techniques  are  useful 
as  checklists,  as  organizational  tools  or  as  a  way 
of  identifying  and  characterizing  problem  areas. 
Matrix  techniques  are  effective  for  tracing  a 
system’s  operational  requirements  through  contrac¬ 
tual  specification  documents,  issues,  and  criteria 
to  sources  of  individual  data  or  specific  test  events. 

13.6.1.4  Dendritic  Analysis  Techniques 

Dendritic  analysis  techniques  are  an  effective  way 
of  decomposing  critical  issues  to  the  point  where 
actual  data  requirements  and  test  measurements 
can  be  identified.  In  these  techniques,  issues  are 
successively  broken  down  into  objectives,  MOEs, 
Measures  of  Performance  (MOPs)  and  data  re¬ 
quirements  in  a  root-like  structure  as  depicted  in 
Figure  13-2.  In  this  approach,  objectives  are  used 
to  clearly  express  the  broad  aspects  of  T&E  re¬ 
lated  to  the  critical  issues  and  the  overall  purpose 
of  the  test.  The  MOEs  are  developed  as  subsets 
of  the  objectives  and  are  designed  to  treat  specific 


and  addressable  parts  of  the  objectives.  Each  MOE 
is  traceable  as  a  direct  contributor  to  one  objec¬ 
tive  and,  through  it,  is  identifiable  as  a  direct  con¬ 
tributor  to  addressing  one  or  more  critical  issues 
(Reference  82).  Each  test  objective  and  MOE  is 
also  linked  to  one  or  more  MOPs  (quantitative  or 
qualitative  measures  of  system  performance  un¬ 
der  specified  conditions)  that,  in  turn,  are  tied  to 
specific  data  elements.  The  dendritic  approach  has 
become  a  standard  military  planning  technique. 

13.6.2  Sources  of  Data 

As  evaluation  and  analysis  planning  matures,  fo¬ 
cus  turns  toward  identifying  data  sources  as  a 
means  for  obtaining  each  data  element.  Initial  iden¬ 
tification  tends  to  be  generic  such  as:  engineering 
study,  simulation,  modeling,  or  contractor  test. 
Later  identification  reflects  specific  studies,  mod¬ 
els  and/or  tests.  A  data  source  matrix  is  a  useful 
planning  tool  to  show  where  data  are  expected  to 
be  obtained  during  the  T&E  of  the  system. 

There  are  many  sources  of  data  that  can  contrib¬ 
ute  to  the  evaluation.  Principal  sources  include: 
studies  and  analyses,  models,  simulations,  war 
games,  contractor  testing.  Development  Test  (DT), 
Operational  Test  (OT),  and  comparable  systems. 

13.7  EVALUATING  DEVELOPMENT 
AND  OPERATIONAL  TESTS 

Technical  and  operational  evaluations  employ  dif¬ 
ferent  techniques  and  have  different  evaluation  cri¬ 
teria.  The  DT&E  is  often  considered  technical 
evaluation  while  OT&E  addresses  the  operational 
aspects  of  a  system.  Technical  evaluation  deals 
primarily  with  instrumented  tests  and  statistically 
valid  data.  An  operational  evaluation  deals  with 
operational  realism  and  the  combat  uncertainties 
(Reference  75).  The  DT&E  uses  technical  crite¬ 
ria  for  evaluating  system  performance.  These  cri¬ 
teria  are  usually  parameters  that  can  be  measured 
during  controlled  DT&E  tests.  They  are  particu¬ 
larly  important  to  the  developing  organization  and 
the  contractor  but  are  of  less  interest  to  the  inde- 
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Figure  13-2.  Dendritic  Approach  to  Test  and  Evaluation 


pendent  operational  tester.  The  operational  tester 
focuses  on  issues  such  as  demonstrating  target  ac¬ 
quisition  at  useful  ranges,  air  superiority  in  com¬ 
bat,  or  the  probability  of  accomplishing  a  given 
mission.  For  example,  during  DT&E,  firing  may 
be  conducted  on  a  round-by-round  basis  with  each 
shot  designed  to  test  an  individual  specification 
or  parameter  with  other  parameters  held  constant. 
Such  testing  is  designed  to  measure  the  technical 
performance  of  the  system.  In  contrast,  in  OT&E 
proper  technical  performance  regarding  individual 
specifications/parameters  is  de-emphasized  and  the 
environment  is  less  controlled.  The  OT&E  author¬ 
ity  must  assess  whether,  given  this  technical  per¬ 
formance,  the  weapon  system  is  operationally 
effective  and  operationally  suitable  when  em¬ 
ployed  under  realistic  combat  (with  opposing 
force)  and  environmental  conditions  by  typical 
personnel. 


The  emphasis  in  DT  is  strictly  on  the  use  of  quan¬ 
titative  data  to  verify  attainment  of  technical  speci¬ 
fications.  Quantitative  data  are  usually  analyzed 
using  some  form  of  statistics.  Qualitative  data  takes 
on  increasing  importance  in  OT&E  when  effec¬ 
tiveness  and  suitability  issues  are  being  explored. 
Many  techniques  are  used  to  analyze  qualitative 
data.  They  range  from  converting  expressions  of 
preference  or  opinion  into  numerical  values  to  es¬ 
tablishing  a  consensus  by  committee.  For  example, 
a  committee  may  assign  values  to  parameters  such 
as  “feel,”  “ease  of  use,”  “friendliness  to  the  user,” 
and  “will  the  user  want  to  use  it,”  on  a  scale  of  1- 
to-10.  Care  should  be  exercised  in  the  interpreta¬ 
tion  of  the  results  of  qualitative  evaluations.  For 
instance,  when  numbers  are  assigned  to  average 
evaluations  and  their  standard  deviations,  mean¬ 
ings  will  differ  from  quantitative  data  averages  and 
standard  deviations. 
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13.7.1  Technical  Evaluation 

The  Services’  materiel  development  organizations 
are  usually  responsible  for  oversight  of  all  aspects 
of  DT&E  including  the  technical  evaluation.  The 
objectives  of  a  technical  evaluation  are: 

•  To  assist  the  developers  by  providing  informa¬ 
tion  relative  to  technical  performance;  qualifi¬ 
cation  of  components;  compatibility,  inter¬ 
operability,  vulnerability,  lethality,  transportabil¬ 
ity,  Reliability,  Availability,  and  Maintainabil¬ 
ity  (RAM);  manpower  and  personnel;  system 
safety;  Integrated  Logistics  Support  (ILS);  cor¬ 
rection  of  deficiencies;  accuracy  of  environ¬ 
mental  documentation;  and  refinement  of 
requirements; 

•  To  ensure  the  effectiveness  of  the  manufactur¬ 
ing  process  of  equipment  and  procedures 
through  production  qualification  T&E; 

•  To  confirm  readiness  for  OT  by  ensuring  that 
the  system  is  stressed  beyond  the  levels  ex¬ 
pected  in  the  OT  environment; 

•  To  provide  information  to  the  decision  author¬ 
ity  at  each  decision  point  regarding  a  system’s 
technical  performance  and  readiness  to  proceed 
to  the  next  phase  of  acquisition; 

•  To  determine  the  system’s  operability  in  the 
required  climatic  and  realistic  battlefield  envi¬ 
ronments  to  include  natural,  induced,  and  coun¬ 
termeasure  environments  (Reference  58). 

13.7.2  Operational  Evaluation 

The  independent  OT&E  authority  is  responsible 
for  the  operational  evaluation.  The  objectives  of 
an  operational  evaluation  are: 


•  To  assist  the  developers  by  providing  informa¬ 
tion  relative  to  operational  performance;  doctrine, 
tactics,  training,  logistics;  safety;  survivability; 
manpower,  technical  publications;  RAM;  correc¬ 
tion  of  deficiencies;  accuracy  of  environmental 
documentation;  and  refinement  of  requirements; 

•  To  assist  decision  makers  ensure  that  only  sys¬ 
tems  that  are  operationally  effective  and  suit¬ 
able  are  delivered  to  the  operating  forces; 

•  To  provide  information  to  the  decision  author¬ 
ity  at  each  decision  point  as  to  a  system’s  op¬ 
erational  effectiveness,  suitability  and  readiness 
to  proceed  to  the  next  phase  of  acquisition; 

•  To  assess,  from  the  user’s  viewpoint,  a  system’s 
desirability,  considering  systems  already  fielded, 
and  the  benefits  or  burdens  associated  with  the 
system  (Reference  83). 

13.8  SUMMARY 

A  primary  consideration  in  identifying  informa¬ 
tion  to  be  generated  by  an  evaluation  program  is 
having  a  clear  understanding  of  the  decisions  the 
information  will  support.  The  importance  of  struc¬ 
turing  the  T&E  program  to  support  the  resolution 
of  critical  issues  cannot  be  overemphasized.  It  is 
the  responsibility  of  those  involved  in  the  evalua¬ 
tion  process  to  ensure  that  the  proper  focus  is 
maintained  on  key  issues,  the  T&E  program  yields 
information  on  critical  technical  and  operational 
issues,  all  data  sources  necessary  for  a  thorough 
evaluation  are  tapped  and  evaluation  results  are 
communicated  in  an  effective  and  timely  manner. 
The  evaluation  process  should  be  evolutionary 
throughout  the  acquisition  phases. 
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MODELING  AND  SIMULATION 
SUPPORT  TO  T&E 


14.1  INTRODUCTION 

This  chapter  discusses  the  applications  of  Model¬ 
ing  and  Simulation  (M&S)  in  Test  and  Evalua¬ 
tion  (T&E).  The  need  for  M&S  has  long  been 
recognized,  as  evidenced  by  this  quotation  from 
the  USAF  Scientific  Advisory  Board  in  June  1965: 

Prediction  of  combat  effectiveness  can  only 
be,  and  therefore  must  be,  made  by  using 
the  test  data  in  analytical  procedures.  This 
analysis  usually  involves  some  type  of 
model,  simulation,  or  game  (i.e.,  the  tools 
of  operations  or  research  analysis).  It  is  the 
exception  and  rarely,  that  the  ‘end  result’ 
i.e.,  combat  effectiveness,  can  be  deduced 
directly  from  test  measurements. 

In  mandating  T&E  early  in  the  acquisition  pro¬ 
cess,  Department  of  Defense  Instruction  (DoDI) 
5000.2,  Operation  of  the  Defense  Acquisition  Sys¬ 
tem,  12  May  2003,  encourages  the  use  of  M&S 
as  a  source  of  T&E  data.  For  instance,  the  Ar¬ 
mored  Family  of  Vehicles  program  used  more 
than  60  models,  simulations,  and  other  test  data 
to  support  system  concept  exploration.  The  reli¬ 
ance  on  M&S  by  this  and  other  acquisition  pro¬ 
grams  provides  the  T&E  community  with  valu¬ 
able  information  which  can  increase  confidence 
levels,  decrease  field  test  time  and  costs,  and  pro¬ 
vide  data  for  pretest  prediction  and  post-test  vali¬ 
dation.  The  Defense  Modeling  and  Simulation 
Office  (DMSO),  working  for  the  Director  Defense 
Research  and  Engineering  (DDR&E),  is  devel¬ 
oping  Office  of  the  Secretary  of  Defense  (OSD) 
guidance  on  the  application  of  M&S  to  the  acqui¬ 
sition  process.  The  DMSO  operates  the  Model¬ 
ing  and  Simulation,  Information  Analysis  Center 


(MSIAC)  to  provide  assistance  to  all  DoD  M&S 
users,  including  program  offices  and  the  acquisi¬ 
tion  community  at  large. 

This  chapter  discusses  using  M&S  to  increase  the 
efficiency  of  the  T&E  process,  reduce  time  and 
cost,  provide  otherwise  unattainable  and  immea¬ 
surable  data,  and  provide  more  timely  and  valid 
results. 

14.2  TYPES  OF  MODELS  AND 
SIMULATIONS 

The  term  “modeling  and  simulation”  is  often  as¬ 
sociated  with  huge  digital  computer  simulations; 
but  it  also  includes  manual  and  man-in-the-loop 
war  games,  Hardware-in-the-Loop  (HWIL)  simu¬ 
lations,  test  beds,  hybrid  laboratory  simulators,  and 
prototypes. 

A  mathematical  model  is  an  abstract  representa¬ 
tion  of  a  system  that  provides  a  means  of  devel¬ 
oping  quantitative  performance  requirements  from 
which  candidate  designs  can  be  developed.  Static 
models  are  those  that  depict  conditions  of  state 
while  dynamic  models  depict  conditions  that  vary 
with  time,  such  as  the  action  of  an  autopilot  in 
controlling  an  aircraft.  Simple  dynamic  models  can 
be  solved  analytically,  and  the  results  represented 
graphically. 

According  to  a  former  Director,  Defense  Test 
and  Evaluation  (DDT&E)  (Reference  118), 
simulations  used  in  T&E  can  be  divided  into 
three  categories: 

Constructive  Simulations:  Computer 
simulations  are  strictly  mathematical 


14-1 


representations  of  systems  and  do  not  em¬ 
ploy  any  actual  hardware.  They  may,  how¬ 
ever,  incorporate  some  of  the  actual  soft¬ 
ware  that  might  be  used  in  a  system.  Early 
in  a  system’s  life  cycle,  computer  simula¬ 
tions  can  be  expected  to  provide  the  most 
system  evaluation  information.  In  many 
cases,  computer  simulations  can  be  readily 
developed  as  modifications  of  existing 
simulations  for  similar  systems.  For  ex¬ 
ample,  successive  generations  of  AIM-7 
missile  simulations  have  been  effectively 
used  in  T&E. 

Virtual  Simulations:  A  system  test  bed 
usually  differs  from  a  computer  simulation 
as  it  contains  some,  but  not  necessarily  all, 
of  the  actual  hardware  that  will  be  a  part 
of  the  system.  Other  elements  of  the  sys¬ 
tem  are  either  not  incorporated  or,  if  they 
are  incorporated,  are  in  the  form  of  com¬ 
puter  simulations.  The  system  operating 
environment  (including  threat)  may  either 
be  physically  simulated,  as  in  the  case  of 
a  flying  test  bed,  or  computer  simulated, 
as  in  the  case  of  a  laboratory  test  bed.  Air¬ 
craft  cockpit  simulators  used  to  evaluate 
pilot  performance  are  good  examples  of 
system  test  beds.  As  development  of  a  sys¬ 
tem  progresses,  more  subsystems  become 
available  in  hardware  form.  These  sub¬ 
systems  can  be  incorporated  into  system 
test  beds  that  typically  provide  a  great  deal 
of  the  system  evaluation  information  used 
during  the  middle  part  of  a  system’s 
development  cycle. 

Another  type  of  virtual  simulation  used  in 
T&E  is  the  system  prototype.  Unlike  the 
system  test  bed,  all  subsystems  are  physi¬ 
cally  incorporated  in  a  system  prototype. 
The  system  prototype  may  closely  repre¬ 
sent  the  final  system  configuration,  de¬ 
pending  on  the  state  of  development  of  the 
various  subsystems  that  compose  it. 
Preproduction  prototype  missiles  and  air¬ 


craft  used  in  operational  testing  by  the  Ser¬ 
vices  are  examples  of  this  class  of  simula¬ 
tion.  As  system  development  proceeds, 
eventually  all  subsystems  will  become 
available  for  incorporation  in  one  or  more 
system  prototypes.  HWIL  simulators  or 
full-up  man-in-the-loop  system  simulators 
may  provide  the  foundation  for  continu¬ 
ous  system  testing  and  improvement. 
These  simulators  can  provide  the  basis  for 
transitioning  hardware  and  software  into 
operationally  realistic  training  devices  with 
mission  rehearsal  capabilities.  Operational 
testing  of  these  prototypes  frequently  pro¬ 
vides  much  of  the  system  evaluation  in¬ 
formation  needed  for  a  decision  on  full- 
scale  production  and  deployment. 

Live  Simulations:  Some  say  that  every¬ 
thing  except  global  combat  is  a  simulation, 
even  limited  regional  engagements.  Live 
exercises  where  troops  use  equipment  un¬ 
der  actual  environmental  conditions  ap¬ 
proaches  real  life  in  combat  while  conduct¬ 
ing  peacetime  operations.  Training  exer¬ 
cises  and  other  live  simulations  provide  a 
testing  ground  with  real  data  on  actual 
hardware,  software,  and  human  perfor¬ 
mance  when  subjected  to  stressful  condi¬ 
tions.  These  data  can  be  used  to  validate 
the  models  and  simulations  used  in  an 
acquisition  program. 

As  illustrated  in  Figure  14-1,  there  is  a  continu¬ 
ous  spectrum  of  simulation  types  with  the  pure 
computer  simulation  at  one  end  and  the  pure  hard¬ 
ware  prototype  at  the  other  end.  As  you  rely  less 
on  live  forces  in  exercises,  the  ability  to  clearly 
understand  the  live  force  exercise’s  representative¬ 
ness  decreases.  At  the  other  end  of  the  spectrum, 
as  you  rely  more  on  virtual  simulations  (software 
generated  activities),  more  of  the  models’  internal 
complexities  and  increasing  complexity  impacts 
the  accuracy  of  results.  Examples  of  uncertainty 
would  be  defining  why  a  battle  was  lost  in  a  war 
game  simulation,  or  a  system  failed  to  perform  as 
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THE  SIMULATION  SPECTRUM 

CRITERIA 

VALUES  CONDUCIVE  TO: 

PHYSICALTESTING 

MODELING  AND  SIMULATION 

TEST  SAMPLE  SIZE/ 

NUMBER  OF  VARIABLES 

SMALL/FEW 

LARGE/MANY 

STATUS  OF  VARIABLES/UNKNOWNS 

CONTROLLABLE 

UNCONTROLLABLE 

PHYSICAL  SIZE  OF  PROBLEM 

SMALLAREA/FEW  PLAYERS 

LARGE  AREA/MANY  PLAYERS 

AVAILABILITY  OF  TEST  EQUIPMENT 

AVAILABLE 

UNAVAILABLE 

AVAILABILITY  OF  TEST  FACILITIES 

RANGES,  OTHER  TEST  AVAILABLE 

BENCHMARKED,  VALIDATED 
COMPUTER  MODELS  AVAILABLE 

TYPES  OF  VARIABLES/UNKNOWNS 

SPATIAL/TERRAIN 

LOW  IMPORTANCE  OF  SPATIAL/ 
TERRAIN 

DIPLOMATIC/POLITICAL  FACTORS 

CONVENTIONAL  CONFLICTS 

NUCLEAR  OR  CHEMICAL  CONFLICTS 

Figure  14-2.  Values  of  Selected  Criteria  Conducive  to  Modeling  and  Simulation 
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expected  in  a  prototype  or  full  system  simulation 
or  why  in  a  flight  simulator,  the  aircraft  seemed  to 
become  inverted  when  crossing  the  equator.  De¬ 
termining  accuracy  is  a  factor  of  the  technical  level 
of  confidence  or  credibility  of  the  simulation. 

14.3  VALIDITY  OF  MODELING  AND 
SIMULATION 

Simulations  are  not  a  substitute  for  live  testing. 
There  are  many  things  that  cannot  be  adequately 
simulated  by  computer  programs;  among  them  are 
the  process  of  decision  and  the  proficiency  of  per¬ 
sonnel  in  the  performance  of  their  functions.  There¬ 
fore,  models  and  simulations  are  not  a  total  sub¬ 
stitution  for  physical  tests  and  evaluations.  Simu¬ 
lations,  manual  and  computer-designed,  can 
complement  and  increase  the  validity  of  live  tests 
and  evaluations  by  proper  selection  and  applica¬ 
tion.  Figure  14-2  contrasts  the  test  criteria  that  are 
conducive  to  modeling  and  simulation  versus 
physical  testing.  Careful  selection  of  the  simula¬ 
tion,  knowledge  of  its  application  and  operation 
and  meticulous  selection  of  input  data  will  pro¬ 
duce  representative  and  valid  results. 

Conversely,  certain  highly  complex  simulations 
can  be  more  effective  analytical  tools  than  limited 
live  testing,  generally  due  to  the  significance  of 
the  external  constraints.  For  example,  nuclear 
weapons  effects  testing  can  only  be  done  in  simu¬ 
lation  because  of  international  treaties.  Testing 
depleted  uranium  ammunition  is  very  expensive 
with  its  related  environmental  considerations.  Fi¬ 
nally,  many  confidence  building  iterations  of 
multi-million  dollar  missile  defense  tests  can  only 
be  executed  using  simulated  events  in  conjunc¬ 
tion  with  a  few  instances  of  real  firings. 

The  important  element  in  using  a  simulation  is  to 
select  one  that  is  representative  and  either  ad¬ 
dresses,  or  is  capable  of  being  modified  to  ad¬ 
dress,  the  level  of  detail  (issues,  thresholds,  and 
objectives)  under  investigation.  Models  and  simu¬ 
lations  must  be  approved  for  use  through  verifi¬ 
cation,  validation,  and  accreditation  processes 


DoD  Directive  (DoDD)  5000.59)  DoD  Model¬ 
ing  and  Simulation  Management,  20  January 
1998,  and  DoDI  5000.61,  DoD  Modeling  and 
Simulation  (M&S)  Verification,  Validation  and 
Accreditation  (W&A),  13  May  2003.  Verification 
is  the  process  of  determining  that  a  model  imple¬ 
mentation  accurately  represents  the  developer’s 
conceptual  description  and  specifications.  Valida¬ 
tion  is  the  process  of  determining:  (a)  the  manner 
and  degree  to  which  a  model  is  an  accurate  repre¬ 
sentation  of  the  real-world  from  the  perspective 
of  the  intended  uses  of  the  model;  and  (b)  the  con¬ 
fidence  that  should  be  placed  on  this  assessment. 
Accreditation  is  the  official  certification  that  a 
model  or  simulation  is  acceptable  for  use  for  a 
specific  purpose. 

14.4  SUPPORT  TO  TEST  DESIGN  AND 
PLANNING 

14.4.1  Modeling  and  Simulation  in  T&E 
Planning 

The  M&S  can  assist  in  the  T&E  planning  process 
and  can  reduce  the  cost  of  testing.  In  Figure  14-3, 
areas  of  particular  application  include  scenario  de¬ 
velopment  and  the  timing  of  test  events;  the  de¬ 
velopment  of  objectives,  essential  elements  of 
analysis,  and  MOEs;  the  identification  of  variables 
for  control  and  measurement;  and  the  development 
of  data  collection,  instrumentation  and  data  analy¬ 
sis  plans.  For  example,  using  simulation,  the  test 
designer  can  examine  system  sensitivities  to 
changes  in  variables  to  determine  the  critical  vari¬ 
ables  and  their  ranges  of  values  to  be  tested.  The 
test  designer  can  also  predict  the  effects  of  vari¬ 
ous  assumptions  and  constraints  and  evaluate  can¬ 
didate  measures  of  effectiveness  to  help  formu¬ 
late  the  test  design.  In  the  live  fire  vulnerability 
testing  of  the  F/A-22  wing  panel,  a  mechanical 
deformation  model  was  used  extensively  in  pre¬ 
test  predictions.  The  pre-test  analysis  was  used  to 
design  the  experiment  that  assisted  in  shot-line 
selection  and  allowed  elimination  of  the  aft  boom 
testing  saving  schedule  and  about  $100,000.  Real 
data  in  post- test  analysis  verified  the  capability  to 


14-4 


FRPDR 


MODELS  &  MS  MS  MS 

SIMULATIONS  A  B  _ C 


CONCEPT  REFINE 

TECH  DEV 

SYSTEM  DEV  &  DEMO 

PRODUCTION  &  DEPLOY 

OPERATIONS  &  SUPPORT 

ANALYSIS 

OF 

ALTERNATIVES 

CAMPAIGN/THEATER 
(OUTCOMES)  \ 

N 

\ 

MISSION/BATTLE 

(EFFECTIVENESS) 

> 

y  UPDATES 

y  UPDATES 

\  UPDATES 
/  ASREQD 

UPDATES 

AS  REQD 

ENGAGEMENT  ' 

(EFFECTIVENESS) 

ENGINEERING  ^ 

(PERFORMANCE) 

/ 

THREAT  SIMULATORS 

SURVIVABILITY/ 

VULNERABILITY 

/ 

VIRTUAL  PROTOTYPES  “ 

PHYSICAL  MODELS 

PHYSICAL  MODELS 

LIVE  PROTOTYPE 

LIVE  PROTOTYPE  TESTING 

LIVE  TESTING 

LIVE  TESTING 

TESTING 

ENGINEERING  (E.G., 
HW/SWIL) 

STIMULATORS 

ENGINEERING  (E.G., 
HW/SWIL  TO  SUPPORT 
PAT&E) 

STIMULATORS 

ENGINEERING  (TO 
SUPPORT  MODIFICATION 
TESTING) 

DISTRIBUTED  SIMULATION- 

PROGRAM  ACTIVITIES/DOCUMENTS 

•  TECHNOLOGY  •PRELIMINARY  •TEMP  *TEMP 

DEVELOPMENT - ►  TEMP  •  DOT&E  REPORT 

•  LFT  REPORT 

•  COMPONENT  DT&E  •  SUBSYSTEM  &  •  DT&E  •  MODIFICATION  TESTING 

SYSTEM  DT&E  ‘PAT&E 


STRATEGY 

•  IDENTIFY  MOEs, 
MOPs 


•  CRITICAL  SYSTEM  #L™E 

CHARACTERISTICS 


•  ID  MODELS  •  EARLY  OPERATIONAL  •  OPERATIONAL 

ASSESSMENT  ASSESSMENT  _ 

•  IOT&E  *  FOT&E  • FOT&E 


Figure  14-3.  Modeling  and  Simulation  Application  in  Test  and  Evaluation 


predict  impulse  damage  within  acceptable  limits, 
resulting  in  greater  use  of  the  model  in  future 
applications. 

Caution  must  be  exercised  when  planning  to  rely 
on  simulations  to  obtain  test  data  as  they  tend  to 
be  expensive  to  develop  or  modify,  difficult  to 
integrate  with  data  from  other  sources,  and  often 
do  not  provide  the  level  of  realism  required  for 
operational  tests.  Although  simulations  are  not  a 
“cure-all,”  they  should  be  explored  and  used  when¬ 
ever  feasible  as  another  source  of  data  for  the 
evaluator  to  consider  during  the  test  evaluation. 
Simulation  fidelity  is  improving  rapidly  with 
enhancements  to  computer  technology  and  it  is 
no  longer  cost  prohibitive  to  develop  credible 
M&S  applications  for  future  weapon  systems. 


Computer  simulations  may  be  used  to  test  the  plan¬ 
ning  for  an  exercise.  By  setting  up  and  running 
the  test  exercise  in  a  simulation,  the  timing  and 
scenario  may  be  tested  and  validated.  Critical 
events  may  include  interaction  of  various  forces 
that  test  the  MOEs  and,  in  turn,  test  objectives. 
Further,  the  simulation  may  be  used  to  verify  the 
statistical  test  design  and  the  instrumentation,  data 
collection,  and  data  analysis  plans.  Essentially,  the 
purpose  of  computer  simulation  in  pretest  plan¬ 
ning  is  to  preview  the  test  to  evaluate  ways  to 
make  test  results  more  effective.  Pretesting  at¬ 
tempts  to  optimize  test  results  by  pointing  out  po¬ 
tential  trouble  spots.  It  constitutes  a  test  setup 
analysis,  which  can  encompass  a  multitude  of  ar¬ 
eas.  The  model-test-model  process  is  an  integrated 
approach  to  using  models  and  simulations  in  sup¬ 
port  of  pre-test  analysis  and  planning;  conducting 
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the  actual  test  and  collecting  data;  and  post-test 
analysis  of  test  results  along  with  further  valida¬ 
tion  of  the  models  using  the  test  data. 

As  an  example  of  simulations  used  in  test  plan¬ 
ning,  consider  a  model  that  portrays  aircraft  ver¬ 
sus  air  defenses.  The  model  can  be  used  to  repli¬ 
cate  typical  scenarios  and  provide  data  on  the 
number  of  engagements,  air  defense  systems  in¬ 
volved,  aircraft  target,  length  and  quality  of  the 
engagement,  and  a  rough  approximation  of  the 
success  of  the  mission  (i.e.,  if  the  aircraft  made  it 
to  the  target).  With  such  data  available,  a  data 
collection  plan  can  be  developed  to  specify,  in 
more  detail,  when  and  where  data  should  be  col¬ 
lected,  from  which  systems,  and  in  what  quantity. 
The  results  of  this  analysis  impact  heavily  on  long 
lead-time  items  such  as  data  collection  devices  and 
data  processing  systems.  The  more  specificity 
available,  the  fewer  the  number  of  surprises  that 
will  occur  downstream.  As  tactics  are  decided 
upon  and  typical  flight  paths  are  generated  for  the 
scenario,  an  analysis  can  be  prepared  on  the  flight 
paths  over  the  terrain  in  question;  and  a  determi¬ 


nation  can  be  made  regarding  whether  the  exist¬ 
ing  instrumentation  can  track  the  numbers  of  air¬ 
craft  involved  in  their  maneuvering  envelopes. 
Alternative  site  arrangements  can  be  examined  and 
tradeoffs  can  be  made  between  the  amount  of 
equipment  to  be  purchased  and  the  types  of  pro¬ 
files  that  can  be  tracked  for  this  particular  test. 
Use  of  such  a  model  can  also  highlight  numerous 
choices  available  to  the  threat  air  defense  system 
in  terms  of  opportunities  for  engagement  and  prac¬ 
tical  applications  of  doctrine  to  the  specific 
situations. 

14.4.2  Simulation,  Test  and  Evaluation 
Process  (STEP) 

The  STEP  has  been  proposed  in  DoD  guidance 
of  the  recent  past  and  is  still  a  valid  concept.  In 
STEP,  simulation  and  test  are  integrated,  each  de¬ 
pending  on  the  other  to  be  effective  and  efficient. 
Simulations  provide  predictions  of  the  system’s 
performance  and  effectiveness,  while  tests  are  part 
of  a  strategy  to  provide  information  regarding  risk 
and  risk  mitigation,  to  provide  empirical  data  to 


Figure  14-4.  STEP  Process 
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validate  models  and  simulations,  and  to  determine 
whether  systems  are  operationally  effective,  suit¬ 
able,  and  survivable  for  intended  use.  A  by-product 
of  this  process  is  a  set  of  models  and  simulations 
with  a  known  degree  of  credibility  providing  the 
potential  for  reuse  in  other  efforts  (Figure  14-4). 

STEP  is  driven  by  mission  and  system  require¬ 
ments.  The  product  of  STEP  is  information.  The 
information  supports  acquisition  program  decisions 
regarding  technical  risk,  performance,  system 
maturity,  operational  effectiveness,  suitability,  and 
survivability.  STEP  applies  to  all  acquisition  pro¬ 
grams,  especially  Major  Defense  Acquisition  Pro¬ 
grams  (MDAPs)  and  Major  Automated  Informa¬ 
tion  Systems  (MAISs). 

Throughout  STEP,  tests  are  conducted  to  collect 
data  for  evaluating  the  system  and  refining  and 
validating  models.  Through  the  model-test-model 
iterative  approach,  the  sets  of  models  mature,  cul¬ 
minating  in  accurate  representations  of  the  system 
with  appropriate  fidelity  which  can  be  used  to  pre¬ 
dict  system  performance  and  to  support  the  ac¬ 
quisition  and  potentially  the  training  communities. 

1.  STEP  begins  with  the  Missions  Needs 
Analysis  and  continues  thorough  the  life 
cycle.  Top  level  requirements  are  used  to 
develop  alternative  concepts  and  select/de¬ 
velop  digital  models  that  are  used  to  evalu¬ 
ate  theater/campaign  and  mission/battle-level 
simulations.  Mission/battle  level  models  are 
used  to  evaluate  the  ability  of  a  multiple  plat¬ 
form  force  package  to  perform  a  specific  mis¬ 
sion.  Mission  and  functional  requirements 
continue  to  be  refined,  and  the  system 
reaches  the  preliminary  design  stage. 

2  Modeling  and  simulation  is  used  both  as  a 
predictive  tool  and  with  test  in  an  iterative 
process  to  evaluate  the  system  design.  The 
consequences  of  design  changes  are  evalu¬ 
ated  and  help  translate  the  most  promising 
design  approach  into  a  stable,  interoperable, 
and  cost  effective  design. 


3.  System  components  and  subsystems  are 
tested  in  a  laboratory  environment.  Data 
from  this  hardware  is  employed  in  the  model- 
test-model  process.  Modeling  and  Simula¬ 
tion  is  used  in  the  planning  of  tests  to  sup¬ 
port  a  more  efficient  use  of  resources.  Simu¬ 
lated  tests  can  be  run  on  virtual  ranges  to 
conduct  rehearsals  and  determine  if  test 
limitations  can  be  resolved.  STEP  tools  are 
used  to  provide  data  for  determining  the  real 
component  or  subsystem’s  performance  and 
interaction  with  other  components.  Model¬ 
ing  and  simulation  is  used  during  both  de¬ 
velopmental  testing  (DT)  and  operational 
testing  (OT)  to  increase  the  amount  of  data 
and  supplement  the  live  test  events  that  are 
needed  to  meet  test  objectives. 

4.  Periodically  throughout  the  acquisition  pro¬ 
cess  the  current  version  of  the  system  under 
development  should  be  reexamined  in  a  syn¬ 
thetic  operational  context  to  reassess  its  mili¬ 
tary  worth.  This  is  one  of  the  significant  as¬ 
pects  of  STEP,  understanding  the  answer  to 
the  question:  What  difference  does  this 
change  make  in  the  system’s  performance? 

5.  STEP  does  not  end  with  fielding  and  de¬ 
ployment  of  a  system,  but  continues  to  the 
end  of  the  system’s  life  cycle.  STEP  results 
in  a  thoroughly  tested  system  with  perfor¬ 
mance  and  suitability  risks  identified.  A  by¬ 
product  is  a  set  of  models  and  simulations 
with  a  known  degree  of  credibility  with  the 
potential  for  reuse  in  other  efforts.  New  test 
data  can  be  applied  to  models  to  incorpo¬ 
rate  any  system  enhancements  and  further 
validate  its  models. 

14.5  SUPPORT  TO  TEST  EXECUTION 

Simulations  can  be  useful  in  test  execution  and 
dynamic  planning.  With  funds  and  other  restric¬ 
tions  limiting  the  number  of  times  that  a  test  may 
be  repeated  and  each  test  conducted  over  several 
days,  it  is  mandatory  that  the  test  director  exercises 
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close  control  over  the  conduct  of  the  test  to  en¬ 
sure  the  specific  types  and  quantities  of  data 
needed  to  meet  the  test  objectives  are  being  gath¬ 
ered  and  to  ensure  adequate  safety.  The  test  di¬ 
rector  must  be  able  to  make  minor  modifications 
to  the  test  plan  and  scenario  to  force  achievement 
of  these  goals.  This  calls  for  a  dynamic  (quick- 
look)  analysis  capability  and  a  dynamic  planning 
capability.  Simulations  may  contribute  to  this  ca¬ 
pability.  For  example,  using  the  same  simulation(s) 
as  used  in  pre-test  planning,  the  tester  could  input 
data  gathered  during  the  first  day  of  the  exercise 
to  determine  the  adequacy  of  the  data  to  fulfill  the 
test  objectives.  Using  this  data,  the  entire  test  could 
be  simulated.  Projected  inadequacies  could  be  iso¬ 
lated,  and  the  test  plans  could  be  modified  to  mini¬ 
mize  the  deficiencies. 

Simulations  may  also  be  used  to  support  test  con¬ 
trol  and  to  ensure  safety.  For  example,  during  mis¬ 
sile  test  firings  at  White  Sands  Missile  Range 
(WSMR),  aerodynamic  simulations  of  the  pro¬ 
posed  test  were  run  on  a  computer  during  actual 
firings  so  that  real-time  missile  position  data  could 
be  compared  continuously  to  the  simulated  mis¬ 
sile  position  data.  If  any  significant  variations  oc¬ 
curred  and  if  the  range  safety  officer  was  too  slow 
(both  types  of  position  data  were  displayed  on 
plotting  boards),  the  computer  issued  a  destruct 
command. 

Simulations  can  be  used  to  augment  tests  by  simu¬ 
lating  non-testable  events  and  scenarios.  Although 
operational  testing  should  be  accomplished  in  as 
realistic  an  operational  environment  as  possible, 
pragmatically  some  environments  are  impossible 
to  simulate  for  safety  or  other  reasons.  Some  of 
these  include  the  environment  of  a  nuclear  battle¬ 
field,  to  include  the  effects  of  nuclear  bursts  on 
friendly  and  enemy  elements.  Others  include  two- 
sided  live  firings  and  adequate  representation  of 
other  forces  to  ascertain  compatibility  and  interop¬ 
erability  data.  Instrumentation,  data  collection  and 
data  reduction  of  large  combined  armed  forces 
(e.g.,  brigade,  division  and  larger-sized  forces) 
become  extremely  difficult  and  costly.  Simulations 


are  not  restricted  by  safety  factors  and  can  realis¬ 
tically  replicate  many  environments  that  are  oth¬ 
erwise  unachievable  in  an  operational  test  and 
evaluation  (OT&E)  —  nuclear  effects,  large  com¬ 
bined  forces,  Electronic  Countermeasures  (ECM), 
Electronic  Counter-Countermeasures  (ECCM)  and 
many  engagements.  These  effects  can  be  simu¬ 
lated  repeatedly  with  no  environmental  impacts, 
only  costing  for  the  simulation  operations. 

Usually,  insufficient  units  are  available  to  simu¬ 
late  the  organizational  relationships  and  interac¬ 
tion  of  the  equipment  with  its  operational  envi¬ 
ronment,  particularly  during  the  early  OT&E  con¬ 
ducted  using  prototype  or  pilot  production-type 
equipment.  Simulations  are  not  constrained  by 
these  limitations.  Data  obtained  from  a  limited  test 
can  be  plugged  into  a  simulation  that  is  capable 
of  handling  many  of  the  types  of  equipment  be¬ 
ing  tested.  It  can  interface  them  with  other  ele¬ 
ments  of  the  blue  forces  and  operate  them  against 
large  elements  of  the  red  forces  to  obtain  interactions. 

End-item  simulators  can  be  used  to  evaluate  de¬ 
sign  characteristics  of  equipment  and  can  be  used 
to  augment  the  results  obtained  using  prototype 
or  pilot  production-type  equipment  that  is  repre¬ 
sentative  of  the  final  item.  The  simulator  may  be 
used  to  expand  test  data  to  obtain  the  required 
iterations  or  to  indicate  that  the  human  interface 
with  the  prototype  equipment  will  not  satisfy  the 
design  requirements. 

It  is  often  necessary  to  use  substitute  or  surrogate 
equipment  in  testing;  e.g.,  American  equipment  is 
used  to  represent  threat-force  equipment.  In  some 
cases  the  substitute  equipment  may  have  greater 
capabilities  than  the  real  equipment,  in  other  cases 
it  may  have  less.  Simulations  are  capable  of  rep¬ 
resenting  the  real  characteristics  of  equipment  and, 
therefore,  can  be  used  as  a  means  of  modifying 
raw  data  collected  during  the  test  to  reflect  real 
characteristics. 

As  an  example,  if  the  substitute  equipment  is  an 
AAA  gun  with  a  tracking  rate  of  30  degrees  per 
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second  and  the  equipment  for  which  it  is  substi¬ 
tuted  has  a  tracking  rate  of  45  degrees  per  sec¬ 
ond,  the  computer  simulation  could  be  used  to 
augment  the  collected,  measured  data  by  deter¬ 
mining  how  many  rounds  could  have  been  fired 
against  each  target  or  whether  targets  that  were 
missed  because  the  tracking  rate  was  too  slow 
could  have  been  engaged  by  the  actual  equipment. 
Consideration  of  other  differing  factors  simul¬ 
taneously  could  have  a  positive  or  negative 
synergistic  effect  on  test  results,  and  could  allow 
evaluation  more  quickly  through  a  credible  simu¬ 
lation  designed  with  the  flexibility  to  replicate 
operational  variations. 

14.6  SUPPORT  TO  ANALYSIS  AND  TEST 
REPORTING 

Modeling  and  simulation  may  be  used  in  post-test 
analysis  to  extend  and  generalize  results  and  to 
extrapolate  to  other  conditions.  The  difficulty  of 
instrumentation  and  controlling  large  exercises  and 
collecting  and  reducing  the  data  and  resource 
costs,  to  some  degree,  limits  the  size  of  T&E.  This 
makes  the  process  of  determining  the  suitability 
of  equipment  to  include  compatibility,  interoper¬ 
ability,  organization,  etc.,  a  difficult  one.  To  a  large 
degree  the  limited  interactions,  interrelationships 
and  compatibility  of  large  forces  may  be  supple¬ 
mented  by  using  actual  data  collected  during  the 
test  and  applying  it  in  the  simulation. 

Simulations  can  be  used  to  extend  test  results,  save 
considerable  energy  (fuel  and  manpower),  and 
save  money  by  reducing  the  need  to  repeat  data 
points  to  improve  the  statistical  sample  or  to  de¬ 
termine  overlooked  or  directly  unmeasured  param¬ 
eters.  Sensitivity  analyses  can  be  run  using  simula¬ 
tions  to  evaluate  the  robustness  of  the  design. 

In  analyzing  the  test  results,  data  can  be  compared 
to  the  results  predicted  by  the  simulations  used 
early  in  the  planning  process.  Thus,  the  simula¬ 
tion  is  validated  by  the  actual  live  test  results,  but 
the  test  results  are  also  validated  by  the  simulation. 


14.7  SIMULATION  INTEGRATION 

Simulations  have  become  so  capable  and  wide¬ 
spread  in  their  application,  and  the  ability  to  net¬ 
work  dissimilar  and/or  geographically  separate 
simulators  in  real  time  to  synergistically  simulate 
more  than  ever  before,  that  whole  new  applica¬ 
tions  are  being  discovered.  Simulations  are  no 
longer  stove-piped  tools  in  distinct  areas,  but  are 
increasingly  crossing  disciplines  and  different  uses. 
The  F-35  Joint  Strike  Fighter  (JSF)  program  uses 
nearly  200  different  models  and  simulations 
throughout  its  many  development  and  testing  ac¬ 
tivities,  in  both  government  centers  and  contrac¬ 
tor  facilities.  Increasingly,  operational  issues  are 
being  determined  through  operational  analysis 
simulations,  which  will  subsequently  impact  OT 
much  later  in  a  program.  Elements  of  the  T&E 
community  should  be  involved  earlier  and  earlier 
in  every  program  to  insure  such  results  are  com¬ 
patible  with  the  OT  objectives  and  requirements. 
For  example,  The  F-35  program  networked  8  dif¬ 
ferent  war  game  simulations  together  to  develop 
the  initial  operational  requirements  which  subse¬ 
quently  became  documented  in  their  Operational 
Requirements  Document  (ORD)  for  the  many  dif¬ 
ferent  required  Service  applications,  i.e.  Navy,  Air 
Force,  Marine  Corps  and  British  Royal  Navy.  Only 
through  this  extensive  “Virtual  Strike  Warfare 
Environment”  were  the  broad  and  concurrent 
range  of  operational  requirements  able  to  be  evalu¬ 
ated  and  established.  New  capabilities  were 
achieved,  but  the  extensive  multi-simulation  net¬ 
work  demanded  new  techniques  for  establishing 
the  technical  confidence  in  the  results. 

14.8  SIMULATION  PLANNING 

With  M&S  becoming  increasing  more  complex, 
more  expensive  and  more  extensive,  the  neces¬ 
sity  to  plan  for  their  use,  and  their  subsequent 
technical  confidence,  must  be  thoroughly  planned 
for.  Testers  are  expected  to  be  involved  earlier  and 
earlier,  including  the  operational  test  agencies,  if 
the  subsequent  T&E  results  are  to  be  accepted. 
Simulation  Support  Plans  (SSP)  are  program 
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documents  which  span  the  many  simulations,  their 
purpose  and  their  expected  credibility.  The  Army, 
Air  Force  and  Marine  Corps  have  policies  requir¬ 
ing  the  early  establishment  of  an  SSP  process,  and 
stand  alone  SSP  documents.  Usually  this  starts 
with  a  program  office  level  simulation  support 
group  of  Service  M&S  experts  who  advise  the 
program  on  M&S  opportunities,  establish  the 
VV&A  process  and  document  these  procedures 
in  the  SSP,  which  extends  across  the  life  cycle  of 
the  system  development.,  testing  and  employment. 
It  is  vital  that  the  SSP  be  fully  coordinated  with 
the  Test  and  Evaluation  Master  Plan  (TEMP). 
Without  early  planning  of  what  each  model  or 
simulation  is  for,  their  expense,  impact  on  system 
design  as  well  as  testing,  earlier  programs  have 
ended  up  with  a  volume  of  different  data  for  dif¬ 


ferent  purposes  which  couldn’t  be  analyzed,  were 
subsequently  found  not  to  be  credible,  or  delayed 
testing  or  program  development. 

14,9  SUMMARY 

M&S  in  T&E  can  be  used  for  concept  evalua¬ 
tion,  extrapolation,  isolation  of  design  effects, 
efficiency,  representation  of  complex  environ¬ 
ments,  and  overcoming  inherent  limitations  in  ac¬ 
tual  testing.  The  use  of  M&S  can  validate  test  re¬ 
sults,  increase  confidence  levels,  reduce  test  costs, 
and  provide  opportunities  to  shorten  the  overall 
acquisition  cycle  by  providing  more  data  earlier 
for  the  decision-maker.  But  it  does  take  time  and 
funding  to  bring  models  and  simulations  along  to 
the  point  that  they  are  useful  during  an  acquisition. 
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TEST  RESOURCES 


15.1  INTRODUCTION 

This  chapter  describes  the  various  types  of  re¬ 
sources  available  for  testing,  explains  test  resource 
planning  in  the  Services,  and  discusses  the  ways 
in  which  test  resources  are  funded. 

According  to  the  Defense  Acquisition  Guidebook, 
the  term  “test  resources”  is  a  collective  term  that 
encompasses  elements  necessary  to  plan,  conduct, 
collect,  and  analyze  data  from  a  test  event  or  pro¬ 
gram.  These  elements  include:  funding  (to  develop 
new  resources  or  use  existing  ones),  manpower 
for  test  conduct  and  support,  test  articles,  models, 
simulations,  threat  simulators,  surrogates,  replicas, 
test-beds,  special  instrumentation,  test  sites,  targets, 
tracking  and  data  acquisition  instrumentation, 
equipment  (for  data  reduction,  communications, 
meteorology,  utilities,  photography,  calibration, 
security,  recovery,  maintenance,  and  repair),  fre¬ 
quency  management  and  control,  and  base/facility 
support  services.  “Testing  planning  and  conduct 
shall  take  full  advantage  of  existing  investment  in 
DoD  [Department  of  Defense]  ranges,  facilities, 
and  other  resources,  wherever  practical,  unless  oth¬ 
erwise  justified  in  the  Test  and  Evaluation  Master 
Plan  [TEMP].” 

Key  DoD  test  resources  are  in  great  demand  by 
competing  acquisition  programs.  Often  special, 
unique,  or  one-of-a-kind  test  resources  must  be 
developed  specifically  for  the  test  program.  It  is 
imperative  that  the  requirements  for  these  test  re¬ 
sources  be  identified  early  in  the  acquisition  pro¬ 
cess  so  adequate  funding  can  be  allotted  for  their 
development,  and  they  will  be  available  when  the 
test  is  scheduled. 


15.2  OBTAINING  TEST  RESOURCES 

15.2.1  Identify  Test  Resources  and 
Instrumentation 

As  early  as  possible,  but  not  later  than  program 
start,  the  test  facilities  and  instrumentation  require¬ 
ments  to  conduct  program  Tests  and  Evaluations 
(T&Es)  should  be  identified  and  a  tentative  sched¬ 
ule  of  test  activities  prepared.  This  information  is 
recorded  in  the  TEMP  and  Service  test  resource 
documentation. 

15.2.2  Require  Multi-Service  OT&E 

Multi-Service  Operational  Test  and  Evaluation 
(MOT&E)  should  be  considered  for  weapon  sys¬ 
tems  requiring  new  operational  concepts  involv¬ 
ing  other  Services.  If  multi-Service  testing  is  used, 
an  analysis  of  the  impact  of  demonstration  on  time 
and  resources  needed  to  execute  the  multi-Service 
tests  should  be  conducted  before  the  low  rate 
production  decision. 

15.2.3  Military  Construction  Program 
Facilities 

Some  programs  cannot  be  tested  without  Military 
Construction  Program  (MCP)  facilities.  To  con¬ 
struct  these  facilities  will  require  long  lead  times; 
therefore,  early  planning  must  be  done  to  ensure 
that  the  facilities  will  be  ready  when  required. 

15.2.4  Test  Sample  Size 

The  primary  basis  for  the  test-sample  size  is  usu¬ 
ally  based  on  one  or  more  of  the  following: 


•  Analysis  of  test  objectives; 

•  Statistical  significance  of  test  results  at  some 
specified  confidence  level; 

•  Availability  of  test  vehicles,  items,  etc.; 

•  Support  resources  or  facilities  available; 

•  Time  available  for  the  test  program. 

15.2.5  Test  Termination 

One  should  not  hesitate  to  terminate  a  test  before 
its  completion  if  it  becomes  clear  that  the  main 
objective  of  the  test  has  already  been  achieved,  is 
unachievable  (due  to  hardware  failure,  unavail¬ 
ability  of  resources,  etc.)  or  if  additional  samples 
will  not  change  the  outcome  and  conclusions  of 
the  test. 

15.2.6  Budget  for  Test 

The  Acquisition  Strategy,  TEMP,  and  budgeting 
documents  should  be  reviewed  regularly  to  en¬ 
sure  that  there  are  adequate  identified  testing  funds 
relative  to  development  and  fabrication  funds. 

The  Acquisition  Strategy,  TEMP,  and  budgeting 
documents  need  careful  scmtiny  to  ensure  that  there 
are  adequate  contingency  funds  to  cover  correc¬ 
tion  of  difficulties  at  a  level  that  matches  industry/ 
government  experience  on  the  contract.  (Testing  to 
correct  deficiencies  found  during  testing,  without 
sufficient  funding  for  proper  correction,  results  in 
Band-Aid®  approaches,  which  require  corrections 
at  a  later  and  more-expensive  time  period.) 

15.2.7  Test  Articles 

A  summary  of  important  test  planning  items  that 
were  identified  by  the  Defense  Science  Board 
(DSB)  is  provided  below: 

•  Ensure  that  the  whole  system,  including  the 
system  user  personnel,  is  tested.  Realistically 


test  the  complete  system,  including  hardware, 
software,  people,  and  all  interfaces.  Get  user 
involved  from  the  start  and  understand  user 
limitations; 

•  Ascertain  that  sufficient  time  and  test  articles  are 
planned.  When  the  technology  is  stressed,  the 
higher  risks  require  more  test  articles  and  time; 

•  In  general,  parts,  subsystems,  and  systems 
should  be  proven  in  that  order  before  incorpo¬ 
rating  them  into  the  next  higher  assembly  for 
more  complete  tests.  The  instrumentation  should 
be  planned  to  permit  diagnosis  of  trouble; 

•  Major  tests  should  never  be  repeated  without 
an  analysis  of  failure  and  corrective  action. 
Allow  for  delays  of  this  nature. 

15.2.8  Major  Range  and  Test  Facility  Base 
(MRTFB) 

The  National  Defense  Authorization  Act  (NDAA) 
of  Fiscal  Year  (FY)2003  directed  the  DoD  estab¬ 
lish  a  Test  Resource  Management  Center 
(DTRMC),  the  director  (three  star  or  senior  civil¬ 
ian)  reporting  directly  to  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logis¬ 
tics  (USD(AT&L)).  The  DTRMC  provides  over¬ 
sight  for  T&E  strategic  planning  and  budgets  for 
the  Department’s  T&E  activities,  including  the 
MRTFB,  Central  Test  and  Evaluation  Investment 
Program  (CTEIP),  and  the  T&E  Science  and 
Technology  (S&T)  Program. 

All  Services  operate  ranges  and  test  facilities  for 
test,  evaluation,  and  training  purposes.  These  ac¬ 
tivities  constitute  the  DoD  Major  Range  and  Test 
Facility  Base  (DoD  Directive  (DoDD)  3200.11, 
1  May  2002).  This  MRTFB  is  described  as  “a 
national  asset  which  shall  be  sized,  operated,  and 
maintained  primarily  for  DoD  T&E  support  mis¬ 
sions,  but  also  is  available  to  all  users  having  a 
valid  requirement  for  its  capabilities.  The  MRTFB 
consists  of  a  broad  base  of  T&E  activities  man¬ 
aged  and  operated  under  uniform  guidelines  to 
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Figure  15-1.  DoD  Major  Range  and  Test  Facility  Base 


provide  T&E  support  to  DoD  Components  re¬ 
sponsible  for  developing  or  operating  materiel  and 
weapon  systems  ”  (Reference  20A).  The  list  of 
MRTFB  activities  and  their  locations  are  shown 
on  Figure  15-1.  Summaries  of  the  capabilities  of 
each  of  these  activities  (with  points  of  contact  listed 
for  further  information)  may  be  found  in  DoD 
3200.1 1-D,  Major  Range  and  Test  Facility  Base 
Summary  of  Capabilities,  June  1983. 

The  MRTFB  facilities  are  available  for  use  by  all 
the  Services,  other  U.S.  government  agencies  and, 
in  certain  cases,  allied  foreign  governments  and 
contractor  organizations.  Scheduling  is  based  on 
a  priority  system;  and  costs  for  usage  are  billed 
uniformly,  as  stated  in  DoDD  3200.11.  The  Di¬ 
rector,  Test  Ranges  Management  Center  sets 
policy  for  the  composition,  use,  and  test  program 
assignments  of  the  MRTFB.  In  turn,  the  individual 
Services  must  fund,  manage,  and  operate  their  ac¬ 
tivities.  They  are  reimbursed  for  direct  costs  by 
each  user  of  the  activity.  The  DTRMC  sponsors 
a  Joint  Test  Assets  Database  which  lists  MRTFB 
and  Operational  Test  Agency  (OTA)  test  facilities, 


test  area  and  range  data,  instrumentation,  and  test 
systems. 

The  DoD  components  wishing  to  use  an  MRTFB 
activity  must  provide  timely  and  complete  notifi¬ 
cation  of  their  requirements,  such  as  special  in¬ 
strumentation  or  ground-support  equipment  re¬ 
quirements,  to  the  particular  activity  using  the 
documentation  formats  prescribed  by  Document 
501-84,  Universal  Documentation  System  Hand¬ 
book,  issued  by  the  Range  Commanders  Council 
(RCC).  The  requirements  must  be  stated  in  the 
TEMP  discussed  below.  Personnel  at  the  MRTFB 
activity  will  coordinate  with  and  assist  prospec¬ 
tive  users  with  their  T&E  planning,  to  include  con¬ 
ducting  trade-off  analyses  and  test  scenario  opti¬ 
mization  based  on  test  objectives  and  test  support 
capabilities. 

15.2.9  Central  Test  and  Evaluation 
Investment  Program  (CTEIP) 

In  1994  the  Principal  Deputy  Under  Secretary  of 
Defense  for  Acquisition  and  Technology 
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(PDUSD(A&T))  directed  that  the  Director,  Test, 
Systems  Engineering,  and  Evaluation  (DTSE&E) 
establish  and  chair  a  steering  group  that  would  over¬ 
see  the  acquisition  and  integration  of  all  training 
and  associated  test  range  instrumentation  and  de¬ 
velop  related  policy.  As  a  result  of  the  reorganiza¬ 
tion  of  the  Office  of  the  Secretary  of  Defense 
(OSD)  T&E  functions  the  Director,  Operational 
Test  and  Evaluation  (DOT&E)  and  subsequently 
the  DTRMC  subsequently  manages  the  implemen¬ 
tation  of  the  Joint  Training  and  Test  Range 
Roadmap  and  executes  the  CTEIP.  The  CTEIP  pro¬ 
vides  OSD  funding  and  a  mechanism  for  the  de¬ 
velopment  and  acquisition  of  new  test  capabilities 
to  satisfy  multi-Service  testing  requirements. 

15,2.10  Service  Test  Facilities 

There  are  other  test  resources  available  besides 
MRTFB.  Frequently  National  Guard  or  Reserve 
units,  commercial  or  international  test  facilities  and 
war  reserve  assets  are  available  to  support  DoD 
T&E.  The  tester  can  determine  resources  avail¬ 
able  by  contacting  his/her  Service  headquarters 
staff  element.  Within  the  Army,  consult  documents 
such  as:  Army  Test  and  Evaluation  Command’s 
(ATEC’s)  database  on  existing  Army  major  test 
facilities,  major  instrumentation,  and  test  equip¬ 
ment;  the  Project  Manager  for  Instrumentation, 
Targets,  and  Threat  Simulators  (PM,  ITTS)  Tar¬ 
gets  Information  Manual  for  a  descriptive  cata¬ 
logue  of  Army  targets  and  foreign  ground  assets 
available  (or  in  development)  for  support  of  T&E 
and  training;  and,  PM  ITTS’  Threat  Inventory 
Database  on  available  assets  for  both  hardware 
simulators  and  software  simulation  systems.  In¬ 
formation  on  specific  Navy  test  resources  is  found 
in  user  manuals  published  by  each  range  and  the 
Commander  Operational  Test  and  Evaluation 
Force  (COMOPTEVFOR)  catalog  of  available 
support. 

15.3  TEST  RESOURCE  PLANNING 

The  development  of  special  test  resources  to 
support  a  weapon  system  test  can  be  costly  and 


time-consuming.  This,  coupled  with  the  competi¬ 
tion  for  existing  test  resources  and  facilities,  re¬ 
quires  that  early  planning  be  accomplished  to  de¬ 
termine  all  test  resource  requirements  for  weapon 
system  T&E.  The  tester  must  use  government  fa¬ 
cilities  whenever  possible  instead  of  funding  con¬ 
struction  of  contractor  test  capabilities. 

Problems  associated  with  range  and  facility  plan¬ 
ning  are  that  major  systems  tend  to  get  top  prior¬ 
ity;  i.e.,  B-1B,  M-l,  etc.  Range  schedules  are  of¬ 
ten  in  conflict  due  to  system  problems,  which 
cause  schedule  delays  during  testing;  and  there  is 
often  a  shortage  of  funds  to  complete  testing. 

15.3.1  TEMP  Resource  Requirements 

The  Program  Manager  (PM)  must  state  all  key 
test  resource  requirements  in  the  TEMP  and  must 
include  items  such  as  unique  instrumentation,  threat 
simulators,  surrogates,  targets  and  test  articles.  In¬ 
cluded  in  the  TEMP  are  a  critical  analysis  of  an¬ 
ticipated  resource  shortfalls,  their  effect  on  sys¬ 
tem  T&E  and  plans  to  correct  resource  deficien¬ 
cies.  As  the  first  TEMP  must  be  prepared  for  pro¬ 
gram  initiation,  initial  test  resource  planning  must 
be  accomplished  very  early.  Refinements  and  re¬ 
assessments  of  test  resource  requirements  are  in¬ 
cluded  in  each  TEMP  update.  The  guidance  for 
the  content  of  the  test  resource  summary  (Part  V) 
of  the  TEMP  is  in  Appendix  2  -  Test  and  Evalu¬ 
ation  Master  Plan,  Defense  Acquisition  Guidebook 
(Table  15-1).  Once  identified,  the  PM  must  then 
work  within  the  Service  headquarters  and  range 
management  structure  to  assure  the  assets  are 
available  when  needed. 

15.3.2  Service  Test  Resource  Planning 

More-detailed  listings  of  required  test  resources  are 
generated  in  conjunction  with  the  detailed  test  plans 
written  by  the  materiel  developer  and  operational 
tester.  These  test  plans  describe  test  objectives. 
Measures  of  Effectiveness  (MOEs),  test  scenarios 
and  specific  test  resource  requirements. 
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Table  15-1. TEMP  Test  Resource  Summary  Section 


PARTV  -TEST  AND  EVALUATION  RESOURCE  SUMMARY 

PROVIDE  A  SUMMARY  (PREFERABLY  IN  ATABLE  OR  MATRIX  FORMAT)  OF  ALL  KEY  TEST  AND  EVALUATION  RESOURCES, 
BOTH  GOVERNMENT  AND  CONTRACTOR,  THAT  WILL  BE  USED  DURING  THE  COURSE  OF  THE  ACQUISITION  PROGRAM. 

THETEMP  SHOULD  PROJECT  THE  KEY  RESOURCES  NECESSARYTO  ACCOMPLISH  DEVELOPMENTALAND  VALIDATION 
TESTING  AND  OPERATIONAL  TEST  AND  EVALUATION.  THE  TEMP  SHOULD  ESTIMATE,  TOTHE  DEGREE  KNOWN  AT 
MILESTONE  B,  THE  KEY  RESOURCES  NECESSARYTO  ACCOMPLISH  DEVELOPMENTAL  TEST  AND  EVALUATION,  LIVE 
FIRE  TEST  AND  EVALUATION,  AND  ALL  OPERATIONAL  TEST  AND  EVALUATION.  THESE  SHOULD  INCLUDE  ELEMENTS  OF 
THE  NATIONAL  TEST  FACILITIES  BASE  (WHICH  INCORPORATES  THE  MAJOR  RANGE  AND  TEST  FACILITY  BASE 
(MRTFB),  CAPABILITIES  DESIGNATED  BY  INDUSTRY  AND  ACADEMIA,  AND  MRTFB  TEST  EQUIPMENT  AND  FACILITIES), 
UNIQUE  INSTRUMENTATION,  THREAT  SIMULATORS,  AND  TARGETS.  AS  SYSTEM  ACQUISITION  PROGRESSES,  THE 
PRELIMINARY  TEST  RESOURCE  REQUIREMENTS  SHALL  BE  REASSESSED  AND  REFINED  AND  SUBSEQUENTTEMP 
UPDATES  SHALL  REFLECT  ANY  CHANGED  SYSTEM  CONCEPTS,  RESOURCE  REQUIREMENTS,  OR  UPDATED  THREAT 
ASSESSMENT.  ANY  RESOURCE  SHORTFALLS  WHICH  INTRODUCE  SIGNIFICANTTEST  LIMITATIONS  SHOULD  BE 
DISCUSSED  WITH  PLANNED  CORRECTIVE  ACTION  OUTLINED.  SPECIFICALLY,  IDENTIFY  THE  FOLLOWING  TEST 
RESOURCES: 


•  TEST  ARTICLES 

•  TEST  SITES  AND  INSTRUMENTATION 

•  TEST  SUPPORT  EQUIPMENT 

•  THREAT  REPRESENTATION 

•  TESTTARGETSAND  EXPENDABLES 

•  OPERATIONAL  FORCE  TEST  SUPPORT 

•  SIMULATORS,  MODELS,  AND  TEST-BEDS 

•  SPECIAL  REQUIREMENTS 

•  TEST  AND  EVALUATION  FUNDING  REQUIREMENTS 

•  MANPOWER/PERSONNELTRAINING 

Source:  Defense  Acquisition  Guidebook. _ 


15.3.2.1  Army  Test  Resource  Planning 

In  the  Army,  the  independent  evaluator,  with  the 
assistance  of  the  developmental  and  operational 
testers,  prepares  input  to  the  TEMP  and  develops 
the  System  Evaluation  Plan  (SEP),  the  primary  plan¬ 
ning  documents  for  integrated  T&E  of  the  system. 
These  documents  should  be  prepared  early  in  the 
acquisition  cycle  (at  the  beginning  of  system  ac¬ 
quisition  activities).  They  describe  the  entire  T&E 
strategy  including  critical  issues,  test  methodology, 
MOEs  and  all  significant  test  resources.  The  TEMP 
and  SEP  provide  the  primary  input  to  the  Outline 
Test  Plan  (OTP),  which  contains  a  detailed  descrip¬ 
tion  of  each  identified  required  test  resource,  where 
and  when  it  is  to  be  provided,  and  the  providing 
organization. 


The  tester  must  coordinate  the  OTP  with  all  ma¬ 
jor  commands  or  agencies  expected  to  provide  test 
resources.  Then,  the  OTP  is  submitted  to  ATEC, 
for  review  by  the  Test  Schedule  and  Review  Com¬ 
mittee  (TSARC)  and  for  incorporation  into  the 
Army’s  Five-Year  Test  Program  (FYTP).  The  ini¬ 
tial  OTP  for  each  test  should  be  submitted  to  the 
TSARC  as  soon  as  testing  is  identified  in  the 
TEMP.  Revised  OTPs  are  submitted  as  more  in¬ 
formation  becomes  available  or  requirements 
change,  but  a  final  comprehensive  version  of  the 
OTP  should  be  submitted  at  least  18  months  be¬ 
fore  the  resources  are  required. 

The  TSARC  is  responsible  for  providing  high- 
level,  centralized  management  of  T&E  resource 
planning.  The  TSARC  is  chaired  by  the 
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Commanding  General  ATEC  and  consists  of  a 
general  officer  or  equivalent  representatives  from 
the  Army  staff  and  major  commands.  The  TSARC 
meets  semiannually  to  review  all  OTPs,  resolve 
conflicts  and  coordinate  all  identified  test  resource 
requirements  for  inclusion  in  the  FYTP.  The  FYTP 
is  a  formal  resource  tasking  document  for  current 
and  near-term  tests  and  a  planning  document  for 
tests  scheduled  for  the  out-years.  All  OTPs  are 
reviewed  during  the  semiannual  reviews  to  en¬ 
sure  that  any  refinements  or  revisions  are  approved 
by  the  TSARC  and  reflected  in  the  FYTP. 

The  TSARC-approved  OTP  is  a  tasking  docu¬ 
ment  by  which  the  tester  requests  Army  test  re¬ 
sources.  The  TSARC  coordinates  resource  re¬ 
quests,  sets  priorities,  resolves  conflicts  and  sched¬ 
ules  resources.  The  resultant  FYTP,  when  ap¬ 
proved  by  the  Headquarters,  Department  of  the 
Army  (HQDA)  Deputy  Chief  of  Staff  for  G-3 
(DCS  G-3),  is  a  formal  tasking  document  that  re¬ 
flects  the  agreements  made  by  the  resource  pro¬ 
viders  (Army  Materiel  Command  (AMC),  Train¬ 
ing  and  Doctrine  Command  (TRADOC),  Forces 
Command  (FORSCOM),  etc.)  to  make  the  re¬ 
quired  test  resources  available  to  the  designated 
tests.  If  test  resources  from  another  Service,  a  non- 
DoD  governmental  agency  (such  as  the  Depart¬ 
ment  of  Energy  (DOE)  or  National  Aeronautics 
and  Space  Administration  (NASA))  or  a  contrac¬ 
tor  are  required,  the  request  is  coordinated  by 
ATEC.  For  example,  the  request  for  a  range  must 
be  made  at  least  two  years  in  advance  to  ensure 
availability.  However,  due  to  the  long  lead  time 
required  to  schedule  these  non-Army  resources, 
their  availability  cannot  be  guaranteed  if  testing  is 
delayed  or  retesting  is  required.  The  use  of  re¬ 
sources  outside  the  U.S.,  such  as  in  Canada,  Ger¬ 
many  or  other  North  Atlantic  Treaty  Organization 
(NATO)  countries,  is  also  handled  by  ATEC. 

15.3.2.2  Navy  Test  Resource  Planning 

In  the  Navy,  the  developing  agency  and  the  op¬ 
erational  tester  are  responsible  for  identifying  the 
specific  test  resources  required  in  testing  the 


weapon  system.  In  developing  requirements  for 
test  resources,  the  PM  and  Operational  Test  Di¬ 
rector  (OTD)  refer  to  documents  such  as  the  Mis¬ 
sion  Need  Statement  (MNS),  Acquisition  Strat¬ 
egy,  Navy  Decision  Coordinating  Paper  (NDCP), 
Operational  Requirement  Document  (ORD),  threat 
assessments,  Secretary  of  the  Navy  Instruction 
(SECNAVINST)  5000.2B,  and  the  Operational 
Test  Director’s  Guide  (Commander,  Operation 
Test  and  Evaluation  Force  (Navy)  (COMOPTEV- 
FOR  Instruction  3960.  ID).  Upon  Chief  of  Naval 
Operations  (CNO)  approval,  the  TEMP  becomes 
the  controlling  management  document  for  all  T&E 
of  the  weapon  system.  It  constitutes  direction  by 
the  CNO  to  conduct  the  T&E  program  defined  in 
the  TEMP,  including  the  commitment  of  Research, 
Development,  Test  and  Evaluation  (RDT&E)  fi¬ 
nancial  support  and  of  fleet  units  and  schedules. 
It  is  prepared  by  the  PM,  who  is  provided  OT&E 
input  by  the  COMOPTEVFOR  OTD.  The  TEMP 
defines  all  T&E  (DT&E,  OT&E  and  Production 
Acceptance  Test  and  Evaluation  (PAT&E))  to  be 
conducted  for  the  system  and  describes,  in  as  much 
detail  as  possible,  the  test  resources  required. 

The  Navy  uses  its  operational  naval  forces  to  pro¬ 
vide  realistic  T&E  of  new  weapon  systems.  Each 
year,  the  CNO  (N-091)  compiles  all  Fleet  support 
requirements  for  RDT&E  program  support  from 
the  TEMPs  and  publishes  the  CNO  Long-Range 
RDT&E  Support  Requirements  document  for  the 
budget  and  out-years.  In  addition,  a  quarterly  fore¬ 
cast  of  support  requirements  is  published  approxi¬ 
mately  five  months  before  the  Fleet  Employment 
Scheduling  Conference  for  the  quarter  in  which 
the  support  is  required.  These  documents  summa¬ 
rize  OT&E  requirements  for  Fleet  services  and  are 
used  by  the  Fleet  for  scheduling  services  and  out- 
year  budget  projections. 

Requests  for  use  of  range  assets  are  usually  initi¬ 
ated  informally  with  a  phone  call  from  the  PM  and/ 
or  OTD  to  the  range  manager  and  followed  by  for¬ 
mal  documentation.  Requests  for  Fleet  support  are 
usually  more  formal.  The  COMOPTEVFOR,  in 
coordination  with  the  PM,  forwards  the  TEMP 
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and  a  Fleet  RDT&E  Support  Request  to  the  CNO. 
Upon  approval  of  the  request,  the  CNO  tasks  the 
Fleet  Commander  by  letter  or  message  to  coordi¬ 
nate  with  OPTEVFOR  to  provide  the  requested 
support. 

Use  of  most  Navy  ranges  must  be  scheduled  at 
least  a  year  in  advance.  Each  range  consolidates 
and  prioritizes  user  requests,  negotiates  conflicts 
and  attempts  to  schedule  range  services  to  satisfy 
all  requests.  If  the  desired  range  services  cannot 
be  made  available  when  required,  the  test  must 
wait;  or  the  CNO  resolves  the  conflict.  Because 
ranges  are  fully  scheduled  in  advance,  it  is  diffi¬ 
cult  to  accommodate  a  test  that  is  delayed  or  re¬ 
quires  additional  range  time  beyond  that  originally 
scheduled.  Again,  the  CNO  can  examine  the  ef¬ 
fects  of  delays  or  retest  requirements  and  issue 
revised  priorities,  as  required. 

Requests  for  use  of  non-Navy  OT&E  resources 
are  initiated  by  COMOPTEVFOR.  The  Opera¬ 
tional  Test  and  Evaluation  Force  (OPTEVFOR) 
is  authorized  direct  liaison  with  other  Service-in¬ 
dependent  OTAs  to  obtain  OTA-controlled  re¬ 
sources.  Requests  for  other  government-owned 
resources  are  forwarded  to  the  CNO  (N-091)  for 
formal  submission  to  the  Service  Chief  (for  Ser¬ 
vice  assets)  or  to  the  appropriate  government 
agency  (e.g.,  DOE  or  NASA).  Use  of  contractor 
resources  is  usually  handled  by  the  PM,  although 
contractor  assets  are  seldom  required  in  OT&E, 
since  the  Fleet  is  used  to  provide  an  operational 
environment.  Requests  for  use  of  foreign  ranges 
are  handled  by  the  N-091  Assistant  for  Interna¬ 
tional  Research  and  Development  (IR&D). 

15.3.2.3  Air  Force  Test  Resource  Planning 

The  test  resources  required  for  T&E  of  an  Air 
Force  weapon  system  are  identified  in  detail  in 
the  Test  Resources  Plan  (TRP),  which  is  prepared 
by  the  responsible  Air  Force  T&E  organization. 
In  general,  the  Air  Force  Operational  Tests  and 
Evaluation  Center  (AFOTEC)  is  the  test  organi¬ 
zation  for  OT&E  programs;  it  obtains  support  from 


a  Service  major  command  test  agency  for  non¬ 
major  programs,  with  AFOTEC  directing  and 
providing  assistance,  as  required. 

During  the  Advanced  Planning  Phase  of  a  weapon 
system  acquisition  (five  to  six  years  before 
OT&E),  AFOTEC  prepares  the  OT&E  section  of 
the  first  full  TRP,  coordinates  the  TRP  with  all 
supporting  organizations  and  assists  the  Resource 
Manager  (RM)  in  programming  required  re¬ 
sources.  The  resource  requirements  listed  in  the 
Resource  Information  Network  TRP  are  devel¬ 
oped  by  the  test  manager,  RM  and  test  support 
group,  using  sources  such  as  the  ORD  and  threat 
assessments.  The  TRP  should  specify,  in  detail, 
all  the  resources  necessary  to  successfully  con¬ 
duct  a  test  when  it  is  entered  in  the  Test  Resource 
Information  Management  System  (TRIMS). 

The  TRP  is  the  formal  means  by  which  test  re¬ 
source  requirements  are  communicated  to  the  Air 
Staff  and  to  the  appropriate  commands  and  agen¬ 
cies  tasked  to  supply  the  needed  resources.  Hence, 
if  a  required  resource  is  not  specified  in  the  TRP, 
it  is  likely  the  resource  will  not  be  available  for 
the  test.  The  TRP  is  revised  and  updated  on  a 
continuous  basis,  since  the  test  resource  require¬ 
ments  become  better  defined  as  the  OT&E  plans 
mature.  The  initial  TRP  serves  as  a  baseline  for 
comparison  of  planned  OT&E  resources  with  ac¬ 
tual  expenditures.  Comparisons  of  the  initial  TRP 
with  subsequent  updates  provide  an  audit  trail  of 
changes  in  the  test  program  and  its  testing  require¬ 
ments.  The  AFOTEC  maintains  all  TRPs  on 
TRIMS;  this  permits  immediate  response  to  all 
queries  regarding  test  resource  requirements. 

The  AFOTEC/RM  consolidates  the  resource  re¬ 
quirements  from  all  TRPs  coordinating  with  par¬ 
ticipating  and  supporting  organizations  and  agen¬ 
cies  outside  AFOTEC.  Twice  yearly,  the  RM  of¬ 
fice  prepares  a  draft  of  the  USAF  Program  for 
Operational  Test  (PO).  The  PO  is  a  master  plan¬ 
ning  and  programming  document  for  resource  re¬ 
quirements  for  all  HQ  USAF-directed  OT&E  and 
is  distributed  to  all  concerned  commands,  agencies 
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and  organizations  for  review  and  coordination.  It 
is  then  submitted  to  the  Air  Staff  for  review  and 
approval  by  the  Operational  Resource  Management 
Assessment  System  for  Test  and  Evaluation 
(ORMAS/TE),  which  operates  under  the  authority 
of  HQ  AF/TE.  The  ORMAS  Board  is  composed 
of  HQ  USAF  action  officers  and  senior  officers 
from  Major  Commands  (MAJCOMs)  and  agen¬ 
cies  involved  in  OT&E;  it  meets  to  resolve  impacts 
and  conflicting  requirements  at  the  appropriate  Air 
Staff  level.  Through  the  ORMAS  process,  HQ 
USAF  approves  the  PO,  which  becomes  a  direc¬ 
tive  to  participants  for  planning,  programming  and 
budgeting  actions.  Agreements  made  among 
ORMAS  participants  regarding  TRP  and  PO  re¬ 
source  requirements  are  considered  binding. 

All  requests  for  test  resources  are  coordinated  by 
HQ  AFOTEC  as  part  of  the  TRP  preparation  pro¬ 
cess.  When  a  new  weapon  system  development 
is  first  identified,  AFOTEC  provides  a  Test  Man¬ 
ager  (TM)  who  begins  long-term  OT&E  planning. 
The  TM  begins  identifying  needed  test  resources, 
such  as  instrumentation,  simulators  and  models, 
and  works  with  the  resources  directorate  to  ob¬ 
tain  them.  If  the  required  resource  does  not  be¬ 
long  to  AFOTEC,  it  will  negotiate  with  the  com¬ 
mands  having  the  resource.  In  the  case  of  models 
and  simulators,  AFOTEC  surveys  what  is  avail¬ 
able,  assesses  credibility,  and  then  coordinates  with 
the  owner  or  developer  to  use  it.  The  Joint  Tech¬ 
nical  Coordinating  Group  publishes  a  document 
on  Electronic  Warfare  (EW)  models. 

Range  scheduling  should  be  done  early.  At  least 
a  year  is  required,  but  often  a  test  can  be  accom¬ 
modated  with  a  few  months’  notice  if  there  is  no 
requirement  for  special  equipment  or  modifications 
to  be  provided  at  the  range.  Some  of  the  Air  Force 
ranges  are  scheduled  well  in  advance  and  cannot 
accommodate  tests  that  encounter  delays  or  retest 
requirements. 

The  RM  attempts  to  resolve  conflicts  among  vari¬ 
ous  systems  competing  for  scarce  test  resources 
and  elevates  the  request  to  the  Commander,  AFO¬ 


TEC,  if  necessary.  Decisions  on  resource  utiliza¬ 
tion  and  scheduling  are  based  on  the  weapon 
system’s  assigned  priority. 

The  resource  manager  and  the  test  manager  also 
arrange  for  use  of  the  resources  of  other  Services, 
non-DoD  government  agencies  and  contractors. 
Use  of  non-U.S.  resources,  such  as  a  Canadian 
range,  are  coordinated  by  Air  Force,  Chief  of 
Staff/Directorate  of  Test  and  Evaluation  (AF/TE) 
and  based  on  formal  Memoranda  of  Understand¬ 
ing  (MOU).  The  U.S.  Air  Force-Europe/Direc- 
torate  of  Operations-Operations  (USAFE/DOQ) 
handles  requests  for  European  ranges.  Use  of  a 
contractor-owned  resource,  such  as  a  model,  is 
often  obtained  through  the  System  Program  Of¬ 
fice  (SPO)  or  a  general  support  contract. 

15.4  TEST  RESOURCE  FUNDING 

The  Future  Years  Defense  Program  (FYDP),  in¬ 
corporating  a  biennial  budgeting  process,  is  the 
basic  DoD  programming  document  that  records, 
summarizes  and  displays  Secretary  of  Defense 
(SECDEF)  decisions.  In  the  FYDP,  costs  are  di¬ 
vided  into  three  categories  for  each  acquisition 
Program  Element  (PE):  R&D  costs,  investment 
costs  and  operating  costs.  The  Congress  appro¬ 
priates  to  the  Office  of  Management  and  Budget 
(OMB),  and  OMB  apportions  funding  through  the 
SECDEF  to  the  Services  and  to  other  defense 
agencies.  The  Services  and  defense  agencies  then 
allocate  funds  to  others  (claimants,  sub-claimants, 
administering  offices,  commanding  generals,  etc.). 

The  Planning,  Programming,  Budgeting  and  Ex¬ 
ecution  (PPBE)  system  is  a  DoD  internal  system 
used  to  develop  input  to  the  Congress  for  each 
year’s  budget  while  developing  future-year  budgets. 
The  PPBE  is  calendar  oriented.  There  are  concur¬ 
rent  two-year  PPBE  cycles  ongoing  at  one  time. 
These  cycles  are:  planning,  programming,  budget¬ 
ing,  and  execution.  At  any  one  time  there  are  three 
budgets  being  worked  by  the  Services.  The  cur¬ 
rent  two-year  budget  is  being  executed.  The  next 
six  years  of  defense  planning  is  being  programmed, 
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and  long-range  program  plans  and  planning 
guidance  are  being  reviewed  for  updating. 

There  are  various  types  of  funding  in  the  PPBE: 
R&D  funding  for  maintaining  the  technology  base; 
exploratory  development  funding  for  conducting 
the  concept  assessments;  advanced  development 
funding  for  conducting  both  the  concept  develop¬ 
ment  and  the  early  prototyping;  engineering  de¬ 
velopment  funding  for  demonstrating  the  Engi¬ 
neering  Development  Model  (EDM);  procurement 
funding  for  conducting  LRIP,  Full  Rate  Produc¬ 
tion  (FRP),  system  deployment  and  operational 
support.  RDT&E  management  and  support  fund¬ 
ing  is  used  throughout  the  development  cycle  until 
the  system  is  operationally  deployed  when  opera¬ 
tions  and  maintenance  (O&M)  funding  is  used.  The 
RDT&E  appropriation  funds  the  costs  associated 
with  R&D  intended  to  improve  performance,  in¬ 
cluding  test  items,  DT&E  and  test  support  of  OT&E 
of  the  system  or  equipment  test  items. 

Funding  that  is  planned,  programmed  and  bud¬ 
geted  through  the  PPBE  cycle  is  not  always  the 
same  funding  amount  that  the  Congress  appropri¬ 
ates  or  the  PM  receives.  If  the  required  funding 
for  a  test  program  is  not  authorized  by  the  Con¬ 
gress,  the  PM  has  four  ways  to  react.  The  PM 
can  submit  a  supplemental  budget  (for  unfunded 
portions  of  the  program),  request  deficiency  fund¬ 
ing  (for  unforeseen  program  problems)  or  use 
transfer  authority  (from  other  programs  within  the 
Service);  or  the  PM  can  try  to  reprogram  the 
needed  funds  (to  restructure  the  program). 

Generally,  testing  that  is  accomplished  for  a  spe¬ 
cific  system  before  the  production  decision  is 
funded  from  RDT&E  appropriations;  and  testing 
that  is  accomplished  after  the  production  decision 
is  funded  from  other  procurement  or  operations 
and  maintenance  appropriations.  Testing  of  Prod¬ 
uct  Improvements  (Pis),  block  upgrades  and  ma¬ 
jor  modifications  is  funded  from  the  same  appro¬ 
priations  as  the  program  development.  Follow-on 
Test  and  Evaluations  (FOT&E)  are  normally 
funded  from  O&M  funds. 


Funding  associated  with  T&E  (including  instrumen¬ 
tation,  targets  and  simulations)  are  identified  in  the 
system  acquisition  cost  estimates,  Service  acquisi¬ 
tion  plans  and  the  TEMP.  General  funding  informa¬ 
tion  for  development  and  operational  tests  follows: 

Development  Test  (DT)  Funding.  Funds  re¬ 
quired  for  conduct  of  engineering  and  develop¬ 
ment  tests  are  programmed  and  budgeted  by  the 
materiel  developer,  based  upon  the  requirements 
of  the  TEMP.  These  costs  may  include,  but  are 
not  limited  to,  procuring  test  samples/prototypes; 
support  equipment;  transportation  costs;  technical 
data;  training  of  test  personnel;  repair  parts;  and 
test-specific  instrumentation,  equipment  and  facili¬ 
ties.  The  DT&E  funds  are  expended  for  contrac¬ 
tor  and  government  developmental  test  activities. 

The  Service  PM  may  be  required  to  pay  for  the 
use  of  test  resources,  such  as  the  MRTFB,  and 
for  the  development  of  specialized  resources 
needed  specifically  for  testing  the  weapon  system 
being  developed. 

Operational  Test  (OT)  Funding.  Funds  required 
to  conduct  OT  are  usually  programmed  and  bud¬ 
geted  by  the  Service  PM  organization.  The  funds 
are  programmed  in  the  Service’s  long-range  test 
program,  and  the  funds  requirements  are  obtained 
from  the  test  resourcing  documentation  and  TEMP. 

15.4.1  Army  Funding 

Test  resources  are  developed  and  funded  under 
various  Army  appropriations.  The  AMC  and  its 
commodity  commands  provide  test  items,  spare 
parts,  support  items  (such  as  diagnostic  equipment) 
and  ammunition.  Soldiers,  ranges,  fuel,  test  sup¬ 
port  personnel  and  maneuver  areas  are  provided 
by  the  TRADOC  or  FORSCOM.  The  weapon 
system  PM  uses  RDT&E  funds  to  reimburse  these 
supporting  commands  for  costs  directly  related  to 
his  test.  The  weapon  system  materiel  developer  is 
also  responsible  for  funding  the  development  of 
new  test  resources  specifically  needed  to  test  the 
weapon  system.  Examples  of  such  special-purpose 
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resources  include  models,  simulations,  special  in¬ 
strumentation  and  test  equipment,  range  modifi¬ 
cations,  EW  simulators  and,  sometimes,  threat 
simulators.  Although  the  Army  has  a  separate  bud¬ 
get  and  development  plan  for  threat  simulators, 
the  PM  ITTS  threat  simulators  program,  many 
weapon  system  developers  still  have  to  fund  the 
cost  of  new  threat  systems  that  are  specifically 
needed  to  test  their  weapon  system.  Funding  for 
Army  operational  testing  is  through  the  PM’s  PE 
and  is  given  to  ATEC  for  direct  control  of  funds 
for  each  program.  Funding  requirements  are  de¬ 
veloped  in  consonance  with  the  Outline  Test  Plan. 

15.4.2  Navy  Funding 

In  the  Navy,  the  weapon  system  PM  is  respon¬ 
sible  for  funding  the  development  of  all  required 
test-specific  resources  from  the  program’s  RDT&E 
funds.  These  resources  include  test  articles,  ex¬ 
pendables,  one-of-a-kind  targets,  data  collection/ 
reduction  and  instrumentation.  The  development 
of  generic  test  resources  that  can  be  used  in  OT&E 
of  multiple  weapon  systems  such  as  targets,  threat 
simulators  and  range  capabilities,  is  funded  from 
the  Office  of  the  Chief  of  Naval  Operations 
(OPNAV)  generic  accounts  (such  as  target  devel¬ 
opment)  and  not  from  weapon  systems  RDT&E. 
The  PM’s  RDT&E  funds  pay  for  all  DT  and  OT 
through  OPEVAL.  The  PM  pays  for  all  post-pro¬ 
duction  OT  with  program  funds. 

15.4.3  Air  Force  Funding 

In  the  Air  Force,  direct-cost  funding  requires  that 
test-peculiar  (direct)  costs  associated  with  a  par¬ 
ticular  test  program  be  reimbursed  by  the  SPO  to 
the  designated  test  agency.  The  RDT&E  appro¬ 
priation  funds  the  cost  associated  with  R&D,  in¬ 
cluding  test  items,  DT&E  and  Air  Force  Materiel 
Command  (AFMC)  support  of  OT&E  of  the 
system  or  equipment  and  the  test  items.  Costs 


associated  with  Initial  Operational  Test  and  Evalu¬ 
ation  (IOT&E)  are  RDT&E  funded,  and  costs  of 
Qualification  Operational  Test  and  Evaluation 
(QOT&E)  are  O&M  funded.  The  AFOTEC  is 
funded  through  its  own  PE  and  has  direct  control 
of  OT&E  funds  for  all  programs.  The  IOT&E 
manager  prepares  a  TRP  that  summarizes  the  re¬ 
source  requirements  for  IOT&E  and  related  test 
support.  All  pretest  IOT&E  planning  is  budgeted 
through  and  paid  out  of  the  O&M  appropriation. 
The  FOT&E  costs  are  paid  by  AFOTEC  and/or 
the  MAJCOM  operating  the  system  and  funded 
by  the  O&M  appropriation. 

15.5  SUMMARY 

Test  resources  have  many  conflicting  demands  and 
their  use  must  be  scheduled  well  in  advance  of  a 
test.  Resources  specific  to  a  particular  test  must 
often  be  developed  and  funded  from  the  PM’s  own 
RDT&E  budget.  Thus,  the  PM  and  his  testers  must 
ensure  that  test  resource  requirements  are  identi¬ 
fied  early  in  the  acquisition  cycle,  that  they  are 
documented  in  the  initial  TEMP,  and  that  modifi¬ 
cations  and  refinements  are  reported  in  the  TEMP 
updates. 

Funds  for  testing  are  provided  by  congressional 
appropriation  to  the  OMB,  which  apportions  the 
funds  to  the  Services  through  the  SECDEF.  The 
PPBE  is  the  DoD  process  used  to  formulate  bud¬ 
get  requests  to  the  Congress.  Requests  by  PMs 
for  test  resources  are  usually  outlined  in  the  TEMP. 
Generally,  system  development  is  funded  from 
RDT&E  funds  until  the  system  is  operationally 
deployed  and  maintained.  O&M  funds  are  used 
for  FOT&E  and  system  maintenance.  The  weapon 
system  materiel  developer  is  also  responsible  for 
funding  the  development  of  new  test  resources 
specifically  needed  to  test  the  weapon  system.  The 
Air  Force  OTA  develops  and  directly  controls 
OT&E  funds. 
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TEST  AND  EVALUATION 
MASTER  PLAN 


16.1  INTRODUCTION 

Guidance  contained  in  Department  of  Defense  In¬ 
struction  (DoDI)  5000.2,  Operation  of  the  Defense 
Acquisition  System,  12  May  2003,  and  the  De¬ 
fense  Acquisition  Guidebook  stipulates  that  a  Test 
and  Evaluation  Master  Plan  (TEMP)  format  shall 
be  used  for  all  Acquisition  Category  (ACAT)  I 
and  IA  or  Office  of  the  Secretary  of  Defense 
(OSD)  designated  oversight  acquisition  programs. 
This  reinforces  the  philosophy  that  good  planning 
supports  good  operations.  For  effective  engineer¬ 
ing  development  and  decision-making  processes, 
an  early  Test  and  Evaluation  (T&E)  strategy  in 
the  Technology  Development  Strategy  (TDS)  must 
be  evolved  into  an  overall  integrating  master  plan 
detailing  the  collection  and  evaluation  of  test  data 
on  required  performance  parameters.  Less  than 
ACAT  I  programs  are  encouraged  to  tailor  their 
T&E  strategy  using  the  TEMP  format  as  a  guide. 
The  TEMP  relates  program  schedule,  test  man¬ 
agement  strategy  and  structure,  and  required  re¬ 
sources  to:  Critical  Operational  Issues  (COIs); 
Critical  Technical  Parameters  (CTPs);  Minimum 
Acceptable  Values  (MAVs)  (thresholds);  acquisi¬ 
tion  strategy;  and,  milestone  decision  points.  Feed¬ 
back  about  the  degree  of  system  performance 
maturity  and  its  operational  effectiveness  and  suit¬ 
ability  during  each  phase  is  essential  to  the  suc¬ 
cessful  fielding  of  equipment  that  satisfies  user 
requirements. 

16.2  TEMP  DEVELOPMENT 

The  development  of  system  T&E  strategy  begins 
during  the  Technology  Development  effort  with 
the  test  planning  in  the  TDS  document.  This 
evolves  as  the  system  is  better  defined  and  T&E 


events  are  consolidated  in  the  TEMP.  Effective 
management  of  the  various  test  processes  are  the 
primary  functions  of  a  Program  Management  Of¬ 
fice  (PMO)  T&E  Working-level  Integrated  Prod¬ 
uct  Team  (WIPT).  The  T&E  strategy  is  highly 
contingent  on  early  system  concept(s)  that  are 
deemed  appropriate  for  satisfying  user  require¬ 
ments.  As  outlined  in  DoD  Directive  (DoDD) 
5000.1,  The  Defense  Acquisition  System,  12  May 
2003,  the  priority  for  selecting  a  solution  is: 

(1)  a  non-materiel  solution,  such  as  changes  to 

Doctrine,  Organization,  Training,  Leadership 

and  Education,  Personnel,  and  Facilities 

(DOT_  LPF)  or, 

(2)  the  sequence  of  Materiel  (M)  alternatives  is: 

(a)  The  procurement  or  modification  of  an 
commercially  available  products,  ser¬ 
vices,  and  technologies  from  domestic 
or  international  system,  or  the  develop¬ 
ment  of  dual-use  technologies. 

(b)  The  additional  production  or  modifica¬ 
tion  of  previously  developed  U.S.  and/ 
or  Allied  militaiy  systems  or  equipment. 

(c)  A  cooperative  development  program 
with  one  or  more  Allied  nations. 

(d)  A  new,  joint,  DoD  Component  or  gov¬ 
ernment  agency  development  program, 
or 

(e)  A  new  DoD  Component-unique  devel¬ 
opment  program. 


The  quality  of  the  test  program  may  directly  reflect 
the  level  of  effort  expended  in  its  development  and 
execution.  This  varies  in  direct  relationship  to  the 
management  imposed  by  the  Program  Manager 
(PM)  and,  to  some  extent,  by  the  system  engi¬ 
neer.  The  PM  must  evaluate  the  utility  of  dedi¬ 
cated  T&E  staff  versus  matrix  support  from  the 
development  command.  The  levels  of  intensity  for 
planning  and  executing  T&E  fluctuate  with 
changes  in  phases  of  the  acquisition  process  and 
in  T&E  staff  support,  as  appropriate. 

Early  planning  of  long-range  strategies  can  be 
supported  with  knowledgeable  planning  teams 
(T&E  Integrated  Product  Teams  (IPT))  and  re¬ 
views  by  panels  of  senior  T&E  management  offi¬ 
cials —  “gray  beards.”  As  the  tempo  of  actual  test 
activities  begins  to  build  concept  to  prototype  to 
engineering  development  model  to  pre-Low  Rate 
Initial  Production  (LRIP),  internal  T&E  manage¬ 
ment  staff  is  needed  to  control  the  processes  and 
evaluate  results. 

16.2.1  Program  Management  Office  (PMO) 
Responsibilities 

The  PMO  is  the  focal  point  of  the  development, 
review  and  approval  process  for  the  program 
TEMP.  The  DoD  acquisition  process  requires  a 
TEMP  as  one  of  the  primary  management  strat¬ 
egy  documents  supporting  the  decision  to  start  or 
terminate  development  efforts.  This  task  is  a  “dif¬ 
ficult  do”  prior  to  program  start  since  some  Ser¬ 
vices  do  not  formulate  or  staff  a  PMO  until  for¬ 
mal  program  initiation.  An  additional  complicat¬ 
ing  factor  is  the  nebulous  condition  of  other  pro¬ 
gram  source  documents  (Capability  Development 
Document  (CDD),  Technical  Management  plan¬ 
ning,  Acquisition  Strategy,  System  Threat  Assess¬ 
ment,  Logistics  Support  Planning,  etc.)  that  are 
also  in  early  stages  of  development/updating  for 
the  milestone  review.  Since  the  TEMP  must  con¬ 
form  to  the  acquisition  strategy  and  other  program 
management  documents,  it  frequently  lags  in  the 
development  process  and  does  not  receive  the 
attention  needed  from  PMO  or  matrix  support 


personnel.  PMO  emphasis  on  early  formulation 
of  the  test  planning  teams  (T&E  WIPT)  is  critical 
to  the  successful  development  of  the  program 
TEMP.  These  teams  should  consist  of  the  requi¬ 
site  players  so  a  comprehensive  and  integrated 
strategy  compatible  with  other  engineering  and 
decision-making  processes  is  developed.  The  PMO 
will  find  that  the  number  of  parties  desiring  coor¬ 
dination  on  the  TEMP  far  exceeds  the  “stream¬ 
lined”  approval  process  signatories,  however,  it 
must  be  coordinated.  An  early  start  in  getting 
Service-level  concurrence  is  important  so  the  mile¬ 
stone  decision  document-submission  schedule  can 
be  supported  with  the  draft  and  final  versions  of 
the  TEMP.  Subsequent  updates  do  not  become 
easier,  as  each  acquisition  phase  brings  new  plan¬ 
ning,  coordination  and  testing  requirements. 

16.2.2  T&E  Planning 

Developing  an  overall  strategy  provides  the  frame¬ 
work  for  incorporating  phase-oriented  T&E  ac¬ 
tivities  that  will  facilitate  the  acquisition  process. 
The  T&E  strategy  should  be  consistent  with  the 
program  acquisition  strategy,  identifying  require¬ 
ments  for  contractor  and  government  Development 
Test  and  Evaluation  (DT&E),  interactions  between 
DT&E  and  Operational  Test  and  Evaluation 
(OT&E),  and  provisions  for  the  separate  Initial 
Operational  Test  and  Evaluation  (IOT&E). 

An  evolutionary  acquisition  strategy  will  gener¬ 
ally  include  an  incremental  or  spiral  approach  with 
moderate-  to  low-risk  technologies  that  should  re¬ 
duce  the  intensity  and  duration  of  the  T&E  pro¬ 
gram.  It  does,  however,  include  a  requirement  for 
post-production  test  activities  as  the  system  is  modi¬ 
fied  to  accommodate  previously  unknown  new 
technologies,  new  threats  or  other  performance 
enhancements. 

A  revolutionary  acquisition  strategy  (single  step) 
incorporates  all  the  latest  technologies  in  the  final 
production  configuration  and  is  generally  a  higher- 
risk  approach.  As  the  contractor  works  on  matur¬ 
ing  emerging  technologies,  the  T&E  workload 
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increases  in  direct  proportion  to  the  technology 
risk  and  difficulty  in  fixing  problems.  There  is  a 
much  higher  potential  for  extended  schedules  with 
iterative  test-fix-test  cycles. 

16.2.3  General  T&E  Planning  Issues 

The  Defense  Science  Board  (DSB)  (Reference  40) 
report  presented  guidance  on  T&E  at  two  levels. 
On  a  general  level  it  discussed  a  number  of  issues 
that  were  appropriate  to  all  weapon  acquisition 
programs.  These  issues,  along  with  a  summary 
discussion,  are  given  below. 

16.2.3.1  Effects  of  Test  Requirements  on 
System  Acquisition 

The  acquisition  strategy  for  the  system  should  al¬ 
low  sufficient  time  between  the  end  of  demon¬ 
stration  testing  and  procurement,  as  contracted  with 
limited  production  decisions,  to  allow  flexibility 
for  modification  of  plans  that  will  be  required.  It 
should  ensure  that  sufficient  dollars  are  available 
not  only  to  conduct  T&E  but  to  allow  for  addi¬ 
tional  T&E  that  is  always  required  due  to  failure, 
design  changes,  etc.;  and,  it  should  be  evaluated 
relative  to  constraints  imposed  by: 

•  The  level  of  system  testing  at  various  stages  of 
the  Research,  Development,  Test,  and  Evalua¬ 
tion  (RDT&E)  cycle; 

•  The  number  of  test  items  available  and  the 
schedule  interface  with  other  systems  needed 
in  the  tests,  such  as  aircraft,  electronics,  etc.; 

•  The  support  required  to  assist  in  preparing  for  and 
conducting  tests  and  analyzing  the  test  results; 

•  Being  evaluated  to  minimize  the  so-called  T&E 
gap  caused  by  lack  of  hardware  during  the  test 
phase. 

16.2.3.2  Test  Requirements  and  Restrictions 

Tests  should: 


•  Have  specific  objectives; 

•  List,  in  advance,  actions  to  be  taken  as  a  con¬ 
sequence  of  the  test  results; 

•  Be  instrumented  to  permit  diagnosis  of  the  cause 
of  lack  of  performance  including  random,  design 
induced,  wear  out  and  operator  error  failure; 

•  If  failures  occur,  not  be  repeated  without  a  de¬ 
tailed  analysis  of  the  failure.  (“Most  likely  the 
failure  will  not  go  away.”) 

16.2.3.3  Trouble  Indicators 

Establish  an  early  detection  scheme  to  identify 
program  illness. 

When  a  program  begins  to  have  trouble,  there  are 
indicators  that  will  show  up  during  testing.  Some 
of  these  indicators  are: 

•  A  test  failure; 

•  Any  repetitive  failure; 

•  A  revision  of  schedule  or  incremental  funding 
that  exceeds  the  original  plan; 

•  Any  relaxation  of  the  basic  requirements  such 
as  lower  performance. 

16.2.3.4  Requirement  for  Test  Rehearsals 

Test  rehearsals  should  be  conducted  for  each  new 
phase  of  testing. 

16.2.4  Scheduling 

Specific  issues  associated  with  test  scheduling  are 
listed  below. 

16.2.4.1  Building  Block  Test  Scheduling 

The  design  of  a  set  of  tests  to  demonstrate  feasi¬ 
bility  prior  to  testing  the  system  level  engineering 
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development  model  should  be  used.  This  will  al¬ 
low  early  testing  of  high-technical-risk  items,  and 
subsequent  tests  can  be  incorporated  into  the  hard¬ 
ware  as  the  system  concept  has  been  demonstrated 
as  feasible. 

16.2.4.2  Component  and  Subsystem  Test 
Plans 

Ensure  a  viable  component  and  subsystem  test 
plan.  Studies  show  that  almost  all  component  fail¬ 
ures  will  be  the  kind  that  cannot  be  easily  detected 
or  prevented  in  full  system  testing.  System  failure 
must  be  detected  and  fixed  in  the  component/sub¬ 
system  stage,  as  detecting  and  correcting  failure 
only  at  the  operational  test  level  results  in  high 
cost. 

16.2.4.3  Phasing  of  DT&E  and  IOT&E 

Problems  that  become  apparent  in  operational  test¬ 
ing  can  often  be  evaluated  faster  with  the  instru¬ 
mented  DT&E  hardware.  The  integrated  test  plan 
should  provide  time  and  money  to  investigate  test 
failures  and  eliminate  causes  of  failures  before 
other,  similar  tests  take  place. 

16.2.4.4  Schedule  IOT&E  to  Include  System 
Interfaces  with  Other  Systems 

Whenever  possible,  the  IOT&E/FOT&E  (follow- 
on  operational  test  and  evaluation)  of  a  weapon 
system  should  be  planned  to  include  other  sys¬ 
tems  that  must  have  a  technical  interface  with  the 
new  system.  For  example,  the  missile  should  be 
tested  on  most  of  the  platforms  for  which  they  are 
programmed. 

A  Preplanned  Product  Improvements  (P3I)  or  in¬ 
crement  strategy  is  a  variant  of  the  evolutionary 
development  process  in  which  you  recognize  the 
high-risk  technologies/subsystems  and  put  them  on 
parallel  development  tracks.  The  testing  strategy 
should  anticipate  the  requirements  to  evaluate  P3I 
item  technology  maturity  and  then  test  the  system 
during  the  integration  of  the  additional  capability. 


Advanced  Technology  Demonstrations  (ATD)  or 
Advanced  Concept  Technology  Demonstrations 
(ACTD)  may  provide  early  insights  into  available 
technologies  for  incorporation  into  developmen¬ 
tal  or  mature,  post-prototype  systems.  Using 
proven,  mature  technology  provides  a  lower-risk 
strategy  and  may  significantly  reduce  the  devel¬ 
opment  testing  work  load.  To  assess  and  manage 
risk,  PMs  and  other  acquisition  managers  should 
use  a  variety  of  techniques,  including  technology 
demonstrations,  prototyping,  and  T&E.  The  pro¬ 
cess  for  verifying  contract  performance  and  item 
specifications,  testing  and  evaluation  of  threshold 
performance  requirements  in  the  Operational  Re¬ 
quirements  Document  (ORD),  exit  criteria  or  the 
acquisition  program  baseline  performance  should 
be  addressed  in  the  DT&E  strategy.  The  DT&E 
is  an  iterative  process  starting  at  configuration  item/ 
software  module  levels  and  continuing  through¬ 
out  the  component  integration  into  subassemblies 
and,  finally,  system-level  performance  evaluations. 
OT&E  is  interwoven  into  early  DT&E  for  maxi¬ 
mizing  the  efficient  use  of  test  articles  and  test 
schedules.  However,  OT&E  must  remain  a  dis¬ 
tinct  thread  of  activity  that  does  not  lose  its  iden¬ 
tity  in  the  tapestry  of  test  events.  Planning  for  test 
resources  is  driven  by  the  sequence  and  intensity 
of  Development  Test  (DT)  and  Operational  Test 
(OT)  events.  Resource  coordination  is  an  equally 
arduous  task,  which  frequently  has  lead  times 
equal  to  major  program  development  activities. 
Included  in  the  program  T&E  strategy  should  be 
an  overshadowing  evaluation  plan,  outlining  meth¬ 
odologies,  models,  simulations,  and  test  data  re¬ 
quired  at  periodic  decision  points. 

The  TEMP  should:  (a)  address  critical  human  is¬ 
sues  to  provide  data  to  validate  the  results  of  hu¬ 
man  factors  engineering  analyses;  and  (b)  require 
identification  of  mission  critical  operational  and 
maintenance  tasks. 

A  reliability  growth  (Test,  Analyze,  Fix,  and  Test 
(TAFT))  program  should  be  developed  to  satisfy 
the  reliability  levels  required  at  Full  Rate  Produc¬ 
tion  (FRP).  Reliability  tests  and  demonstrations 
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(DoD  3235. 1-H,  Test  and  Evaluation  of  System 
Reliability  Availability  and  Maintainability,  A 
Primer,  March  1982)  will  be  based  on  actual  or 
simulated  operational  conditions. 

Maintainability  will  be  verified  with  a  maintain¬ 
ability  demonstration  (DoD  3235. 1-H)  before  FRP. 

As  early  as  practicable,  developers  and  test  agen¬ 
cies  will  assess  survivability  and  validate  critical 
survivability  characteristics  at  as  high  a  system 
level  as  possible.  The  TEMP  will  identify  the 
means  by  which  the  survivability  objective  will 
be  validated. 

Field  engineering  test  facilities  and  testing  in  the 
intended  operational  environments  are  required  to: 
(1)  verify  electric  or  electronic  systems  predicted 
performance;  (2)  establish  confidence  in  electro¬ 
magnetic  compatibility  design  based  on  standards 
and  specifications;  and  (3)  validate  electromagnetic 
compatibility  analysis  methodology. 

The  TEMP  will  address  health  hazard  and  safety 
critical  issues  to  provide  data  to  validate  the  re¬ 
sults  of  system  safety  analyses. 

The  TEMP  strategy  should  directly  support  the 
development  of  more-detailed  planning  and  re¬ 
source  documents  needed  to  execute  the  actual 
test  events  and  subsequent  evaluations. 

The  TEMP  shall  provide  a  road  map  for  integrated 
simulation,  test,  and  evaluation  plans,  schedules, 
and  resource  requirements  necessary  to  accomplish 
the  T&E  program.  T&E  planning  should  address 
Measures  of  Effectiveness/Suitability  (MOEs/ 
MOS)  with  appropriate  quantitative  criteria,  test 
event  or  scenario  description,  resource  require¬ 
ments  and  test  limitations.  Test  planning,  at  a  mini¬ 
mum,  must  address  all  system  components  that 
are  critical  to  the  achievement  and  demonstration 
of  contract  technical  performance  specifications 
and  threshold  values  specified  in  the  Capability 
Production  Document  (CPD). 


16.3  TEMP  FORMAT 

The  format  specified  in  the  Defense  Acquisition 
Guidebook,  Appendix  2,  is  required  for  all  acquisi¬ 
tion  category  I,  IA  and  OSD  designated  oversight 
programs  (Table  16-1).  It  may  be  tailored  as  needed 
for  lesser  category  acquisition  programs  at  the  dis¬ 
cretion  of  the  milestone  decision  authority.  The 
TEMP  is  intended  to  be  a  summary  document  out¬ 
lining  DT&E  and  OT&E  management  responsi¬ 
bilities  across  all  phases  of  the  acquisition  process. 
When  the  development  is  a  multi-Service  or  joint 
acquisition  program,  one  integrated  TEMP  is  de¬ 
veloped  with  Service  annexes,  as  required.  A  Cap¬ 
stone  TEMP  may  not  be  appropriate  for  a  single 
major  weapon  platform  but  could  be  used  to  en¬ 
compass  testing  of  a  collection  of  individual  sys¬ 
tems,  each  with  its  own  annex  (e.g.,  Missile  De¬ 
fense  Agency  (MDA),  Family  of  Tactical  Vehicles, 
Future  Combat  Systems  (FCSs)).  A  program  TEMP 
is  updated  at  milestones,  upon  program  baseline 
breach  and  for  other  significant  program  changes. 
Updates  may  consist  of  page  changes  and  are  no 
longer  required  when  a  program  has  no  further 
development  activities. 

The  TEMP  is  a  living  document  that  must  ad¬ 
dress  changes  to  critical  issues  associated  with  an 
acquisition  program.  Major  changes  in  program 
requirements,  schedule,  or  funding  usually  result 
in  a  change  in  the  test  program.  Thus,  the  TEMP 
must  be  reviewed  and  updated  on  program 
change,  on  baseline  breach  and  before  each  mile¬ 
stone  decision,  to  ensure  that  T&E  requirements 
are  current.  As  the  primary  document  used  in  the 
OSD  review  and  milestone  decision  process  to 
assess  the  adequacy  of  planned  testing  and  evalu¬ 
ation,  the  TEMP  must  be  of  sufficient  scope  and 
content  to  explain  the  entire  T&E  program.  The 
key  topics  in  the  TEMP  are  shown  in  Table  16-1. 

Each  TEMP  submitted  to  OSD  should  be  a  sum¬ 
mary  document,  detailed  only  to  the  extent  neces¬ 
sary  to  show  the  rationale  for  the  type,  amount  and 
schedules  of  the  testing  planned.  It  must  relate  the 
T&E  effort  clearly  to  technical  risks,  operational 
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Table  16-1.  Test  and  Evaluation  Master  Plan  Format 


PARTI 

SYSTEM  INTRODUCTION 

MISSION  DESCRIPTION 

SYSTEM  DESCRIPTION 

SYSTEM  THREAT  ASSESSMENT 

MEASURES  OF  EFFECTIVENESS  AND  SUITABILITY 

CRITICAL  TECHNICAL  PARAMETERS 

PART  II 

INTEGRATED  TEST  PROGRAM  SUMMARY 

INTEGRATED  TEST  PROGRAM  SCHEDULE 

MANAGEMENT 

PART  III 

DEVELOPMENTTESTAND  EVALUATION  OUTLINE 

DEVELOPMENTTESTAND  EVALUATION  OVERVIEW 

FUTURE  DEVELOPMENTAL  TEST  AND  EVALUATION  LIMITATIONS 

PART  IV 

OPERATIONAL  TEST  AND  EVALUATION  OUTLINE 

OPERATIONALTEST AND  EVALUATION  OVERVIEW 

CRITICAL  OPERATIONAL  ISSUES 

FUTURE  OPERATIONAL  TEST  AND  EVALUATION  LIMITATIONS 

LIVE  FIRE  TEST  AND  EVALUATION 

PARTV 

TEST  AND  EVALUATION  RESOURCE  SUMMARY 

TEST  ARTICLES 

TEST  SITES  AND  INSTRUMENTATION 

TEST  SUPPORT  EQUIPMENT 

THREAT  REPRESENTATION 

TESTTARGETSAND  EXPENDABLES 

OPERATIONAL  FORCE  TEST  SUPPORT 

SIMULATIONS,  MODELS,  AND  TEST  BEDS 

SPECIAL  REQUIREMENTS 

TEST  AND  EVALUATION  FUNDING  REQUIREMENTS 

MANPOWER/PERSONNEL  TRAINING 

APPENDIX  A 

BIBLIOGRAPHY 

APPENDIX  B 

ACRONYMS 

APPENDIX  C 

POINTS  OF  CONTACT 

ATTACHMENTS  (AS  APPROPRIATE) 

SOURCE:  Defense  Acquisition  Guidebook. 

issues  and  concepts,  system  performance,  reliabil¬ 
ity,  availability,  maintainability,  logistic  objectives 
and  requirements,  and  major  decision  points.  It 
should  summarize  the  testing  accomplished  to  date 
and  explain  the  relationship  of  the  various  models 
and  simulations,  subsystem  tests,  integrated  system 
development  tests  and  initial  operational  tests  that, 
when  analyzed  in  combination,  provide  confidence 
in  the  system’s  readiness  to  proceed  into  the  next 
acquisition  phase.  The  TEMP  must  address  the 
T&E  to  be  accomplished  in  each  program  phase, 
with  the  next  phase  addressed  in  the  most  detail. 


The  TEMP  is  also  used  as  a  coordination  docu¬ 
ment  to  outline  each  test  and  support  organization’s 
role  in  the  T&E  program  and  identify  major  test 
facilities  and  resources.  The  TEMP  supporting  the 
production  and  initial  deployment  decision  must 
include  the  T&E  planned  to  verify  the  correction 
of  deficiencies  and  to  complete  production  qualifi¬ 
cation  testing  and  FOT&E. 

The  objective  of  the  OSD  TEMP  review  process, 
often  using  the  Automated  Test  Planning  System 
software,  is  to  ensure  successful  T&E  programs 
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that  will  support  decisions  to  commit  resources  at 
major  milestones.  Some  of  the  T&E  issues  con¬ 
sidered  during  the  TEMP  review  process  include: 

(1)  Are  DT&E  and  OT&E  initiated  early  to  as¬ 
sess  performance,  identify  risks,  and  estimate 
operational  potential? 

(2)  Are  critical  issues,  test  directives,  and  evalu¬ 
ation  criteria  related  to  mission  need  and 
operational  requirements  established  well 
before  testing  begins? 

(3)  Are  provisions  made  for  collecting  sufficient 
test  data  with  appropriate  test  instrumenta¬ 
tion  to  minimize  subjective  judgment? 

(4)  Is  OT&E  conducted  by  an  organization  in¬ 
dependent  of  the  developer  and  user? 

(5)  Do  the  test  methodology  and  instrumenta¬ 
tion  provide  a  mature  and  flexible  network 


of  resources  that  stress  (as  early  as  possible) 
the  weapon  system  in  a  variety  of  realistic 
environments? 

16.4  SUMMARY 

The  PMO  is  directly  responsible  for  the  content 
and  quality  of  the  test  strategy  and  planning  docu¬ 
ment.  The  TEMP,  as  an  integrated  summary  man¬ 
agement  tool,  requires  an  extensive  commitment 
of  man-hours  and  PM  guidance.  The  interactions 
of  the  various  T&E  players  and  support  agencies 
in  the  T&E  WIPT  must  be  woven  into  the  fabric 
of  the  total  system  acquisition  strategy.  Cost  and 
schedule  implications  must  be  negotiated  to  en¬ 
sure  a  viable  T&E  program  that  provides  timely 
and  accurate  data  to  the  engineering  and  manage¬ 
ment  decision  makers. 
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Many  program  managers  face  several  T&E  issues  that  must  be 
resolved  to  get  their  particular  weapon  system  tested  and  ultimately 
fielded.  These  issues  may  include  modeling  and  simulation  support, 
combined  and  concurrent  testing,  test  resources,  survivability  and 
lethality  testing,  multi-Service  testing,  or  international  T&E.  Each 
issue  presents  a  unique  set  of  challenges  for  the  program  manager 
when  he/she  develops  the  integrated  strategy  for  the  T&E  program. 


SOFTWARE 
SYSTEMS  TESTING 


17.1  INTRODUCTION 

Software  development  presents  a  major  develop¬ 
ment  risk  for  military  weapons  and  National  Se¬ 
curity  Systems  (NSS).  Software  is  found  in  Ma¬ 
jor  Automated  Information  Systems  (MAISs)  and 
weapon  system  software.  High  cost  Software  sys¬ 
tems,  such  as  personnel  records  management  sys¬ 
tems,  financial  accounting  systems,  or  logistics 
records,  that  are  the  end  item  solution  to  user  re¬ 
quirements  fall  in  the  MAIS  category.  Perfor¬ 
mance  requirements  for  the  MAIS  typically  drive 
the  host  hardware  configurations  and  are  managed 
by  the  Information  Technology  Acquisition  Board 
(ITAB)  chaired  by  the  Assistant  Secretary  of  De¬ 
fense  (Command,  Control,  Communications  and 
Intelligence  (ASD(C3I))/Chief  Information  Officer 
(CIO).  The  Director,  Operational  Test  and  Evalu¬ 
ation  (DOT&E)  is  a  principal  member  of  the  ITAB. 
Software  developments,  such  as  avionics  systems, 
weapons  targeting  and  control,  and  navigation 
computers,  that  are  a  subset  of  the  hardware  solu¬ 
tion  to  user  requirements  fall  in  the  weapon  sys¬ 
tem  software  category.  Performance  requirements 
for  the  system  hardware  are  flowed  down  to  drive 
the  functionality  of  the  software  resident  in 
onboard  computers.  The  effectiveness  of  the 
weapon  system  software  is  reviewed  as  part  of 
the  overall  system  review  by  the  Defense  Acqui¬ 
sition  Board  (DAB)  chaired  by  the  Under  Secre¬ 
tary  of  Defense  for  Acquisition  and  Technology 
(USD(A&T)).  The  DOT&E  is  a  principal  mem¬ 
ber  and  the  Deputy  Director,  Development,  Test 
and  Evaluation  (DT&E)  is  an  advisor  to  the  DAB. 
No  Milestone  A,  B,  or  Full-rate  Production  (FRP) 
decision  (or  their  equivalent)  shall  be  granted  for 
a  MAIS  until  the  Department  of  Defense  (DoD) 
CIO  certifies  that  the  MAIS  program  is  being 


developed  in  accordance  with  the  Clinger-Cohen 
Act  provisions.  (DoD  Instruction  (DoDI)  5000.2, 
Operation  of  the  Defense  Acquisition  Program, 
12  May  2003.)  For  MDAP  and  MAIS  programs, 
the  DoD  Component  CIO’s  confirmation  (for 
Major  Defense  Acquisition  Programs  (MDAPs) 
and  certification  (for  MAISs)  shall  be  provided  to 
both  the  DoD  CIO  and  the  Milestone  Decision 
Authority  (MDA). 

Software  development  historically  has  escalated 
the  cost  and  reduced  the  reliability  of  weapon  sys¬ 
tems.  Embedded  computer  systems  that  are  physi¬ 
cally  incorporated  into  larger  weapon  systems, 
have  a  major  function  of  data  processing.  The 
output  of  the  systems  are  normally  information, 
control  signals  or  computer  data  required  by  the 
host  system  to  complete  its  mission.  Although 
hardware  and  software  often  contribute  in  equal 
measure  to  successful  implementation  of  system 
functions,  there  have  been  relative  imbalances  in 
their  treatment  during  system  development. 

Automated  Information  Systems  (AISs),  once  de¬ 
veloped,  integrated  and  tested  in  the  host  hard¬ 
ware  are  essentially  ready  for  production.  Soft¬ 
ware  in  weapon  systems,  once  integrated  in  the 
host  hardware,  continue  to  be  tested  as  a  compo¬ 
nent  of  the  total  system  and  is  not  ready  for  pro¬ 
duction  until  the  total  system  has  successfully  dem¬ 
onstrated  required  performance.  Any  changes  to 
weapon  system  hardware  configuration  may 
stimulate  changes  to  the  software.  The  develop¬ 
ment  of  all  software  systems  involves  a  series  of 
activities  in  which  there  are  frequent  opportuni¬ 
ties  for  errors.  Errors  may  occur  at  the  inception 
of  the  process,  when  the  requirements  may  be  er¬ 
roneously  specified,  or  later  in  the  development 


cycle,  when  system  integration  is  implemented. 
This  chapter  addresses  the  use  of  testing  to  obtain 
insights  into  the  development  risk  of  AIS  and 
weapon  system  software,  particularly  as  it  pertains 
to  the  software  development  processes. 

17.2  DEFINITIONS 

The  term  Automated  Information  System  (AIS)  is 
defined  in  DoD  5000.2-R,  Mandatory  Procedures 
for  Major  Defense  Acquisition  Programs  ( MDAPs) 
and  Major  Automated  Information  System  (MAIS) 
Acquisition  Programs,  June  2001,  as  a  combina¬ 
tion  of  computer  hardware  and  software,  data,  or 
telecommunications,  that  performs  functions  such 
as  collecting,  processing,  transmitting,  and  display¬ 
ing  information.  Excluded  are  computer  resources, 
both  hardware  and  software,  that  are:  physical  part 
of,  dedicated  to,  or  essential  in  real  time  to  the 
mission  performance  of  weapon  systems.  (There 
is  some  indication  that  DoD  Directive  (DoDD) 
8000.1,  Management  of  DoD  Information  Re¬ 
sources  and  Information  Technology,  27  Febru¬ 
ary  2002,  Defense  Information  Management  Pro¬ 
gram  providing  guidance  on  AIS  development,  will 
be  incorporated  in  a  future  change  to  the  5000 
Series  on  acquisition  management.) 

The  term  weapon  system  software  includes  auto¬ 
mated  data  processing  equipment,  software  or  ser¬ 
vices;  and  the  function,  operation  or  use  of  the 
equipment  software  or  services  involves: 

(1)  Intelligence  activities; 

(2)  Cryptologic  activities  related  to  national  se¬ 
curity; 

(3)  Command  and  control  of  military  forces; 

(4)  Equipment  that  is  an  integral  part  of  a  weap¬ 
ons  system; 

(5)  Critical,  direct  fulfillment  of  military  or  in¬ 
telligence  missions. 


Acquisition  of  software  for  DoD  is  described  in 
Military  Standard  (MIL-STD)-498,  Software  De¬ 
velopment  and  Documentation,  5  December 
1994,  rescinded)  which  has  been  waiver  for  use 
until  commercial  standards  such  as  EIA  640  or 
J-Std-016  ( Software  Life  Cycle  Processes,  Soft¬ 
ware  Development,  September  1995)  become  the 
guidance  for  software  development.  Guidance 
may  also  be  found  in  the  Defense  Acquisition 
Guidebook. 

17.3  PURPOSE  OF  SOFTWARE  TEST 
AND  EVALUATION 

A  major  problem  in  software  development  is  a 
lack  of  well-defined  requirements.  If  require¬ 
ments  are  not  well-defined,  errors  can  multiply 
throughout  the  development  process.  As  illus¬ 
trated  in  Figure  17-1,  errors  may  occur  at  the 
inception  of  the  process.  These  errors  may  oc¬ 
cur  during  requirements  definition,  when  objec¬ 
tives  may  be  erroneously  or  imperfectly  speci¬ 
fied;  during  the  later  design  and  development 
stages,  when  these  objectives  are  implemented; 
and  during  software  maintenance  and  operational 
phases,  when  software  changes  are  needed  to 
eliminate  errors  or  enhance  performance.  Esti¬ 
mates  of  increased  software  costs  arising  from 
incomplete  testing  help  to  illustrate  the  dimen¬ 
sion  of  software  Life-Cycle  Costs  (LCCs).  Av¬ 
eraged  over  the  operational  life  cycle  of  a  com¬ 
puter  system,  development  costs  encompass  ap¬ 
proximately  30  percent  of  total  system  costs.  The 
remaining  70  percent  of  LCCs  are  associated  with 
maintenance,  which  includes  system  enhance¬ 
ments  and  error  correction.  Complete  testing  dur¬ 
ing  earlier  development  phases  may  have  detected 
these  errors.  The  relative  costs  of  error  correc¬ 
tion  increase  as  a  function  of  time  from  the  start 
of  the  development  process.  Relative  costs  of 
error  correction  rise  dramatically  between  require¬ 
ments  and  design  phases  and  more  dramatically 
during  code  implementation  (Figure  17-1). 

Previous  research  in  the  area  of  software  Test  and 
Evaluation  (T&E)  reveals  that  half  of  all  maintenance 
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Figure  17-1.  The  Error  Avalanche 


costs  are  incurred  in  the  correction  of  previously 
undetected  errors.  Approximately  one-half  of  the 
operational  LCCs  can  be  traced  directly  to  inad¬ 
equate  or  incomplete  testing  activities.  In  addition 
to  cost  increases,  operational  implications  of  soft¬ 
ware  errors  in  weapon  systems  can  result  in  mis¬ 
sion  critical  software  failures  that  may  impact  mis¬ 
sion  success  and  personnel  safety. 

A  more  systematic  and  rigorous  approach  to  soft¬ 
ware  testing  is  required.  To  be  effective,  this  ap¬ 
proach  must  be  applied  to  all  phases  of  the  devel¬ 
opment  process  in  a  planned  and  coordinated 
manner,  beginning  at  the  earliest  design  stages  and 
proceeding  through  operational  testing  of  the  in¬ 
tegrated  system.  Early,  detailed  software  T&E 
planning  is  critical  to  the  successful  development 
of  a  computer  system. 


17.4  SOFTWARE  DEVELOPMENT 
PROCESS 

Software  engineering  technologies  used  to  pro¬ 
duce  operational  software  are  key  risk  factors  in  a 
development  program.  The  T&E  program  should 
help  determine  which  of  these  technologies  in¬ 
crease  risk  and  have  a  life-cycle  impact.  A  princi¬ 
pal  source  of  risk  is  the  support  software  required 
to  develop  operational  software.  In  terms  of  life- 
cycle  impact,  operational  software  problems  are 
commonly  associated  with  the  difficulty  in  main¬ 
taining  and  supporting  the  software  once  de¬ 
ployed.  Software  assessment  requires  an  analysis 
of  the  life-cycle  impact,  which  varies  depending 
on  the  technology  used  to  design  and  implement 
the  software.  One  approach  to  reducing  long-term 
life-cycle  risks  is  to  use  a  commercial  language 
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and  common  hardware  throughout  the  develop¬ 
ment  and  operation  of  the  software.  These  life- 
cycle  characteristics  that  affect  operational  capa¬ 
bilities  must  be  addressed  in  the  Test  and  Evalua¬ 
tion  Master  Plan  (TEMP),  and  tests  should  be 
developed  to  identify  problems  caused  by  these 
characteristics.  The  technology  used  to  design  and 
implement  the  software  may  significantly  affect 
software  supportability  and  maintainability. 

The  TEMP  must  sufficiently  describe  the  accep¬ 
tance  criteria  or  software  maturity  metrics  from  the 
written  specifications  that  will  lead  to  operational 
effectiveness  and  suitability.  The  specifications  must 
define  the  required  software  metrics  to  set  objec¬ 
tives  and  thresholds  for  mission  critical  functions. 
Additionally,  these  metrics  should  be  evaluated  at 
file  appropriate  stage  of  system  development  rather 
than  at  some  arbitrarily  imposed  milestone. 

17.5  T&E  IN  THE  SOFTWARE  LIFE 
CYCLE 

Software  testing  is  an  iterative  process  executed 
at  all  development  stages  to  examine  program 


design  and  code  to  expose  errors.  Software  test 
planning  should  be  described  in  the  TEMP  with 
the  same  care  as  test  planning  for  other  system 
components  (Figure  17-2,  17-3). 

17.5.1  Testing  Approach 

The  integration  of  software  development  into  the 
overall  acquisition  process  dictates  a  testing  pro¬ 
cess  consistent  with  the  bottom-up  approach  taken 
with  hardware  development.  The  earliest  stage  of 
software  testing  is  characterized  by  heavy  human 
involvement  in  basic  design  and  coding  processes. 
Thus,  human  testing  is  defined  as  informal,  non- 
computer-based  methods  of  evaluating  architec¬ 
tures,  designs  and  interfaces.  It  can  consist  of: 

•  Inspections — The  programmer  explains  his/her 
work  to  a  small  group  of  peers  with  discussion 
and  direct  feedback  on  errors,  inconsistencies  and 
omissions. 

•  Walk-through  —  A  group  of  peers  develop 
test  cases  to  evaluate  work  to  date  and  give 
direct  feedback  to  the  programmer. 
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Figure  17-3.  Spiral  Model  for  AIS  Development  Process 


•  Desk  Checking  —  A  self  evaluation  is  made 
by  the  programmer  of  his/her  own  work.  There 
is  a  low  probability  of  identifying  his/her  er¬ 
rors  of  logic  or  coding. 

•  Peer  Ratings  —  Mutually  supportive,  anony¬ 
mous  reviews  are  performed  by  groups  of  peers 
with  collaborative  evaluations  and  feedback. 

•  Design  Reviews  —  Preliminary  Design  Re¬ 
views  (PDRs)  and  Critical  Design  Reviews 
(CDRs)  provide  milestones  in  the  development 
efforts  that  review  development  and  evaluations 
to  date.  An  Independent  Verification  and  Vali¬ 
dation  (IV&V)  contractor  may  facilitate  the 
government’s  ability  to  give  meaningful  feed¬ 
back. 

Once  the  development  effort  has  matured  beyond 
the  benefits  of  human  testing,  computerized-soft- 
ware-only  testing  may  be  appropriate.  It  is  per¬ 
formed  to  determine  the  functionality  of  the  soft¬ 
ware  when  tested  as  an  entity  or  “build.”  Docu¬ 
mentation  control  is  essential  so  that  test  results 
are  correlated  with  the  appropriate  version  of  the 
build.  Software  testing  is  usually  conducted  using 
some  combination  of  “black  box”  and  “white  box” 
testing. 

•  Black  Box  —  Functional  testing  of  a  software 
unit  without  knowledge  of  how  the  internal 
structure  or  logic  will  process  the  input  to  ob¬ 
tain  the  specified  output.  Within-boundary  and 
out-of-boundary  stimulants  test  the  software’s 
ability  to  handle  abnormal  events.  Most  likely 
cases  are  tested  to  provide  a  reasonable  assur¬ 
ance  that  the  software  will  demonstrate  speci¬ 
fied  performance.  Even  the  simplest  software 
designs  rapidly  exceed  our  capacity  to  test  all 
alternatives. 

•  White  Box  —  Structural  testing  of  the  internal 
logic  and  software  structure  provides  an  oppor¬ 
tunity  for  more  extensive  identification  and  test¬ 
ing  of  critical  paths.  The  process  and  objectives 
are  otherwise  very  similar  to  black  box  testing. 


Testing  should  be  performed  from  the  bottom  up. 
The  smallest  controlled  software  modules  —  com¬ 
puter  software  units  —  are  tested  individually. 
They  are  then  combined  or  integrated  and  tested 
in  larger  aggregate  groups  or  builds.  When  this 
process  is  complete,  the  software  system  is  tested 
in  its  entirety.  Obviously,  as  errors  are  found  in 
the  latter  stages  of  the  test  program,  a  return  to 
earlier  portions  of  the  development  program  to 
provide  corrections  is  required.  The  cost  impact 
of  error  detection  and  correction  can  be  dimin¬ 
ished  using  the  bottom-up  testing  approach. 

System  level  testing  can  begin  once  all  modules 
in  the  Computer  Software  Configuration  Item 
(CSCI)  have  been  coded  and  individually  tested. 
A  Software  Integration  Lab  (SIL),  with  adequate 
machine  time  and  appropriate  simulations,  will 
facilitate  hardware  simulation/emulation  and  the 
operating  environment.  If  data  analysis  indicates 
proper  software  functioning,  it  is  time  to  advance 
to  a  more  complex  and  realistic  test  environment. 

•  Hot  Bench  Testing  —  Integration  of  the  soft¬ 
ware  released  from  the  SIL  for  full-up  testing 
with  actual  system  hardware  in  a  Hardware- 
in-the-Loop  (HWIL)  facility  marks  a  signifi¬ 
cant  advance  in  the  development  process.  Close 
approximation  of  the  actual  operating  environ¬ 
ment  should  provide  test  sequences  and  stress 
needed  to  evaluate  the  effectiveness  of  the  soft¬ 
ware  system(s).  Problems  stimulated  by  the 
“noisy  environment,”  interface  problems,  Elec¬ 
tromagnetic  Interference  (EMI)  and  different 
electrical  transients  should  surface.  Good  hard¬ 
ware  and  software  test  programs  leading  up  to 
HWIL  testing  should  aid  in  isolating  problems 
to  the  hardware  or  software  side  of  the  system. 
Caution  should  be  taken  to  avoid  any  outside 
stimuli  that  might  trigger  unrealistic  responses. 

•  Field  Testing  —  Development  Test  and  Evalu¬ 
ation  (DT&E)  and  Operational  Test  and  Evalu¬ 
ation  (OA  (operational  Assessment),  IOT&E) 
events  must  be  designed  to  provide  for  data 
collection  processes  and  instrumentation  that 
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will  measure  system  responses  and  allow  data 
analysts  to  identify  the  appropriate  causes  of 
malfunctions.  Field  testing  should  be  rigorous, 
providing  environmental  stresses  and  mission 
profiles  likely  to  be  encountered  in  operational 
scenarios.  Government  software  support  facili¬ 
ties  tasked  for  future  maintenance  of  the  soft¬ 
ware  system  should  be  brought  on  board  to 
familiarize  them  with  the  system  operating  char¬ 
acteristics  and  documentation.  Their  expertise 
should  be  included  in  the  software  T&E  pro¬ 
cess  to  assist  in  the  selection  of  stimuli  likely 
to  expose  software  problems. 

It  is  critical  that  adequate  software  T&E  informa¬ 
tion  be  contained  in  documents  such  as  TEMPs 
and  test  plans.  The  TEMP  must  define  character¬ 
istics  of  critical  software  components  that  effec¬ 
tively  address  objectives  and  thresholds  for  mis¬ 
sion  critical  functions.  The  Measures  of  Effective¬ 
ness  (MOEs)  must  support  the  critical  software 
issues.  The  test  plan  should  specify  the  test  meth¬ 
odologies  that  will  be  applied.  Test  methodolo¬ 
gies  consist  of  two  components.  The  first  is  the 
test  strategy  that  guides  the  overall  testing  effort, 
and  the  second  is  the  testing  technique  that  is  ap¬ 
plied  within  the  framework  of  a  test  strategy. 

Effective  test  methodologies  require  realistic  soft¬ 
ware  test  environments  and  scenarios.  The  test 
scenarios  must  be  appropriate  for  the  test  objec¬ 
tives;  i.e.,  the  test  results  must  be  interpretable  in 
terms  of  software  test  objectives.  The  test  scenarios 
and  analysis  should  actually  verify  and  validate 
accomplishment  of  requirements.  Information  as¬ 
surance  testing  must  be  conducted  on  any  system 
that  collects,  stores,  transmits  or  processes  classi¬ 
fied  or  unclassified  information.  In  the  case  of 
Information  Technology  (IT)  systems,  including 
NSS,  DT&E  should  support  the  DoD  Informa¬ 
tion  Technology  Security  Certification  and  Ac¬ 
creditation  Process  (ITSCAP)  and  Joint  Interop¬ 
erability  Certification  (JIC)  (DoDD  4630.5,  In¬ 
teroperability  and  Supportability  of  Information 
Technology  (IT)  and  National  Security  Systems 
(NSS),  5  May  2004,  and  Chairman  of  the  Joint 


Chiefs  of  Staff  Instruction  (CJCSI)  6212.01C, 
Interoperability  and  Supportability  of  Information 
Technology  and  National  Security  Systems,  20 
November  2003)  processes.  (DoDI  5000.2)  The 
test  environments  must  be  chosen  on  a  careful 
analysis  of  characteristics  to  be  demonstrated  and 
its  relationship  to  the  development,  operational  and 
support  environments.  In  addition,  environment 
must  be  representative  of  that  in  which  the  soft¬ 
ware  will  be  maintained.  At  Milestone  C,  for 
MAIS,  the  MDA  shall  approve,  in  coordination 
with  the  DOT&E,  the  quantity  and  location  of  sites 
for  a  limited  deployment  for  IOT&E.  (DoDI 
5000.2) 

17.5.2  Independent  Verification  and 
Validation  (IV&V) 

IV&V  are  risk-reducing  techniques  that  are  ap¬ 
plied  to  major  software  development  efforts.  The 
primary  purpose  of  IV&V  is  to  ensure  that  soft¬ 
ware  meets  requirements  and  is  reliable  and  main¬ 
tainable.  The  IV&V  is  effective  only  if  imple¬ 
mented  early  in  the  software  development  sched¬ 
ule.  Requirements  analysis  and  risk  assessment  are 
the  most  critical  activities  performed  by  IV&V  or¬ 
ganizations;  their  effectiveness  is  limited  if  brought 
on  board  a  project  after  the  fact.  Often,  there  is  a 
reluctance  to  implement  IV&V  because  of  the  costs 
involved,  but  early  implementation  of  IV&V  will 
result  in  lower  overall  costs  of  error  correction  and 
software  maintenance.  As  development  efforts 
progress,  IV&V  involvement  typically  decreases. 
This  is  due  more  to  the  expense  of  continued  in¬ 
volvement  than  to  a  lack  of  need.  For  an  IV&V 
program  to  be  effective,  it  must  be  the  responsi¬ 
bility  of  an  individual  or  organization  external  to 
the  software  development  Program  Manager 
(PM). 

The  application  of  the  IV&V  process  to  software 
development  maximizes  the  maintainability  of  the 
fielded  software  system,  while  minimizing  the  cost 
of  developing  and  fielding  it.  Maintenance  of  a 
software  system  falls  into  several  major  categories: 
corrective  maintenance,  modifying  software  to 
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correct  errors  in  operation;  adaptive  maintenance, 
modifying  the  software  to  meet  changing  re¬ 
quirements;  and  perfective  maintenance,  modi¬ 
fying  the  software  to  incorporate  new  features 
or  improvements. 

The  IV&V  process  maximizes  the  reliability  of 
the  software  product,  which  eases  the  performance 
of  and  minimizes  the  need  for  corrective  mainte¬ 
nance.  It  attempts  to  maximize  the  flexibility  of 
the  software  product,  which  eases  the  performance 
of  adaptive  and  perfective  maintenance.  These 
goals  are  achieved  primarily  by  determining  at  each 
step  of  the  software  development  process  that  the 
software  product  completely  and  correctly  meets 
the  specific  requirements  determined  at  the  previ¬ 
ous  step  of  development.  This  step-by-step,  itera¬ 
tive  process  continues  from  the  initial  definition 
of  system  performance  requirements  through  fi¬ 
nal  acceptance  testing. 

The  review  of  software  documentation  at  each 
stage  of  development  is  a  major  portion  of  the 
verification  process.  The  current  documentation 
is  a  description  of  the  software  product  at  the 
present  stage  of  development  and  will  define  the 
requirements  laid  on  the  software  product  at  the 
following  stage.  Careful  examination  and  analy¬ 
sis  of  the  development  documentation  ensure  that 
each  step  in  the  software  design  process  is  con¬ 
sistent  with  the  previous  step.  Omissions,  incon¬ 
sistencies  or  design  errors  can  then  be  identified 
and  corrected  early  in  the  development  process. 

Continuing  participation  in  formal  and  informal 
design  reviews  by  the  IV&V  organization  main¬ 
tains  the  communication  flow  between  software 
system  “customers”  and  developers,  ensuring  that 
software  design  and  production  proceeds  with 
minimal  delays  and  misunderstandings.  Frequent 
informal  reviews,  design  and  code  walk-through 
and  audits  ensure  that  the  programming  standards, 
software  engineering  standards,  Software  Quality 
Assurance  (SQA)  and  configuration  management 
procedures  designed  to  produce  a  reliable,  main¬ 
tainable  operational  software  system  are  followed 


throughout  the  process.  Continuous  monitoring  of 
computer  hardware  resource  allocation  through¬ 
out  the  software  development  process  also  ensures 
that  the  fielded  system  has  adequate  capacity  to 
meet  operation  and  maintainability  requirements. 

The  entire  testing  process,  from  the  planning  stage 
through  final  acceptance  test,  is  also  approached 
in  a  step-by-step  manner  by  the  IV&V  process. 
At  each  stage  of  development,  the  functional  re¬ 
quirements  determine  test  criteria  as  well  as  de¬ 
sign  criteria  for  the  next  stage.  An  important  func¬ 
tion  of  the  IV&V  process  is  to  ensure  that  the  test 
requirements  are  derived  directly  from  the  perfor¬ 
mance  requirements  and  are  independent  of  de¬ 
sign  implementation.  Monitoring  of,  participation 
in  and  performance  of  the  various  testing  and  in¬ 
spection  activities  by  the  IV&V  contractor  ensure 
that  the  developed  software  meets  requirements 
at  each  stage  of  development. 

Throughout  the  software  development  process,  the 
IV&V  contractor  reviews  any  proposals  for  soft¬ 
ware  enhancement  or  change,  proposed  changes 
in  development  base-  lines,  and  proposed  solu¬ 
tions  to  design  or  implementation  problems  to 
ensure  that  the  original  performance  requirements 
are  not  forgotten.  An  important  facet  of  the  IV&V 
contractor’s  role  is  to  act  as  the  objective  third 
party,  continuously  maintaining  the  “audit  trail” 
from  the  initial  performance  requirements  to  the 
final  operational  system. 

17.6  SUMMARY 

There  is  a  useful  body  of  software  testing  tech¬ 
nologies  that  can  be  applied  to  testing  of  AIS  and 
weapon  system  software.  As  a  technical  discipline, 
though,  software  testing  is  still  maturing.  There  is 
a  growing  foundation  of  guidance  documents  to 
guide  the  PM  in  choosing  one  testing  technique 
over  another.  One  example  is  the  U.S.  Air  Force 
Software  Technology  Support  Center’s  (STSC’s) 
Guidelines  for  Successful  Acquisition  and  Man¬ 
agement  of  Software-intensive  Systems.  The  Air 
Force  Operational  Test  and  Evaluation  Center 
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(AFOTEC)  has  also  developed  a  course  on 
Software  OT&E.  It  is  apparent  that  systematic 
T&E  techniques  are  far  superior  to  ad-hoc  testing 
techniques.  Implementation  of  an  effective  soft¬ 
ware  T&E  plan  requires  a  set  of  strong  technical 
and  management  controls.  Given  the  increasing 


amount  of  AIS  and  weapon  system  software  being 
acquired,  there  will  be  an  increased  emphasis  on 
tools  and  techniques  for  software  T&E.  Another 
resource  is  the  Software  Program  Manager’s  Net¬ 
work  that  publishes  guides  on  Best  Practices  in 
Software  T&E,  Vol.  I  &  II.  ('http://www.spmn.com) 
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TESTING  FOR  VULNERABILITY 
AND  LETHALITY 


18.1  INTRODUCTION 

This  chapter  addresses  the  need  to  explore  the  vul¬ 
nerability  and  lethality  aspects  of  a  system  through 
Test  and  Evaluation  (T&E)  practices  and  proce¬ 
dures.  In  particular,  this  chapter  describes  the  leg¬ 
islatively-mandated  Live  Fire  Test  Program,  which 
has  been  established  to  evaluate  the  survivability 
and  lethality  of  developing  systems.  (Table  18-1) 
It  also  discusses  the  role  of  T&E  in  assessing  a 
system’s  ability  to  perform  in  a  nuclear  combat 
environment.  The  discussion  of  testing  for  nuclear 
survivability  is  based  primarily  on  information  con¬ 
tained  in  the  Nuclear  Survivability  Handbook  for 
OT&E,  prepared  by  the  Air  Force  Operational  Test 
and  Evaluation  Center  (AFOTEC)  (Reference  88). 

18.2  LIVE  FIRE  TESTING 
18.2.1  Background 

In  March  1984,  the  Office  of  the  Secretary  of  De¬ 
fense  (OSD)  chartered  a  joint  T&E  program  des¬ 
ignated  “The  Joint  Live  Fire  Program.”  This  pro¬ 
gram  was  to  assess  the  vulnerabilities  and  lethality’s 
of  selected  U.S.  and  threat  systems  already  fielded. 


The  controversy  over  joint  Live  Fire  Testing  (LFT) 
of  the  Army’s  Bradley  Fighting  Vehicle  System, 
subsequent  congressional  hearings  and  media  ex¬ 
posure  resulted  in  provisions  being  incorporated  in 
the  National  Defense  Authorization  Act  (NDAA) 
of  Fiscal  Year  (FY)87.  This  act  required  an  OSD- 
managed  LFT  program  for  major  acquisition  pro¬ 
grams  fitting  certain  criteria.  Subsequent  amend¬ 
ments  to  legislative  guidance  have  dictated  the  cur¬ 
rent  program.  The  Department  of  Defense  (DoD) 
implementation  of  congressional  guidance  in  De¬ 
fense  Acquisition  Guidebook,  requires  that  “cov¬ 
ered  systems,  major  munitions  programs,  missile 
programs,  or  Product  Improvements  (Pis)  to  these” 
(i.e.,  Acquisition  Category  (ACAT)  I  and  II  pro¬ 
grams)  must  execute  survivability  and  lethality  test¬ 
ing  before  Full-Rate  Production  (FRP).  The  legis¬ 
lation  dealing  with  the  term  “covered  system”  has 
been  clarified  as  a  system  “that  DOT&E,  acting 
for  the  Secretary  of  Defense  [SECDEF],  has  de¬ 
termined  to  be:  a  major  system  within  the  meaning 
of  that  term  in  10  U.S.C.  2302(5)  that  is  (1)  user- 
occupied  and  designed  to  provide  some  degree  of 
protection  to  its  occupants  in  combat;  or  (2)  A  con¬ 
ventional  munitions  program  or  missile  program; 
or  (3)  A  conventional  munitions  program  for  which 


Table  18-1.  Relationships  Between  Key  Concepts 


!  PERSPECTIVE 

|;  TERMINOLOGY 

DEFENSIVE 

OFFENSIVE 

MEANING 

SURVIVABILITY 

EFFECTIVENESS 

X 

X 

PROBABILITY  OF  ENGAGEMENT 

VULNERABILITY 

LETHALITY 

X 

X 

PROBABILITY  OF  KILL  GIVEN  A  HIT 

SUSCEPTIBILITY 

X 

PROBABILITY  OF  ENGAGEMENT 

Source:  Adapted  from  Live  Fire  Testing:  Evaluating  DoD’s  Programs, 

U.S.  General  Accounting  Office,  GAO/PEMD-87-17,  August  1987,  page  15. 

more  than  1,000,000  rounds  are  planned  to  be  ac¬ 
quired;  or  (4)  A  modification  to  a  covered  system 
that  is  likely  to  affect  significantly  the  survivabil¬ 
ity  or  lethality  of  such  a  system.” 


been  incorporated  into  the  latest  revision  to  the 
5000  Series  Defense  Acquisition  Guidebook. 

18.2.2  Live  Fire  Tests 


The  Secretary  of  Defense  (SECDEF)  has  delegated 
the  authority  to  waive  requirements  for  the  full-up, 
system  level  Live  Fire  Test  and  Evaluation 
(LFT&E)  before  the  system  passes  the  program 
initiation  milestone,  to  the  Under  Secretary  of  De¬ 
fense  (Acquisition  and  Technology  [USD(AT&L)] 
for  ACAT  ID  and  the  CAE  for  ACAT II  programs, 
when  it  would  be  unreasonably  expensive  and  im¬ 
practical.  An  alternative  vulnerability  and  lethality 
T&E  program  must  still  be  accomplished.  Programs 
subject  to  LFT  or  designated  for  oversight  are  listed 
on  the  OSD  annual  T&E  oversight  list.  The  DoD 
agent  for  management  of  the  Live  Fire  Test  pro¬ 
gram  is  the  Director,  Operational  Test  and  Evalua¬ 
tion  (DOT&E).  This  type  of  Development  Test  and 
Evaluation  (DT&E)  must  be  planned  to  start  early 
enough  in  the  development  process  to  impact  de¬ 
sign  and  to  provide  timely  test  data  for  the  OSD 
Live  Fire  Test  Report  required  for  the  Full  Rate 
Production  Decision  Review  (FRPDR)  and  con¬ 
gressional  committees.  The  Service-detailed  Live 
Fire  Test  Plan  must  be  reviewed  and  approved  by 
the  DOT&E,  and  LFT  must  be  addressed  in  Part 
IV  of  the  program  Test  and  Evaluation  Master  Plan 
(TEMP).  The  OSD  had  previously  published 
guidelines,  elements  of  which  have  subsequently 


There  are  varying  types  and  degrees  of  live  fire 
tests.  The  matrix  in  Table  18-2  illustrates  the  vari¬ 
ous  possible  combinations.  Full-scale,  full-up  test¬ 
ing  is  usually  considered  to  be  the  most  realistic 
and  is  the  type  of  testing  called  for  in  the  NDAA 
for  FY87. 

The  importance  of  full-scale  testing  has  been  well 
demonstrated  by  the  Joint  Live  Fire  (JLF)  tests. 
In  one  case,  these  tests  contradicted  earlier  con¬ 
clusions  concerning  the  flammability  of  a  new 
hydraulic  fluid  used  in  F-15  and  F-16  aircraft. 
Laboratory  tests  had  demonstrated  that  the  new 
fluid  was  less  flammable  than  the  standard  fluid. 
However,  during  the  JLF  tests,  30  percent  of  the 
shots  on  the  new  fluid  resulted  in  fires  contrasted 
with  15  percent  of  the  shots  on  the  standard  fluid 
(Reference  17). 

While  much  insight  and  valuable  wisdom  are  to 
be  obtained  through  the  testing  of  components  or 
subsystems,  some  phenomena  are  only  observable 
when  full-up  systems  are  tested.  The  interaction 
of  such  phenomena  has  been  termed  “cascading 
damage.”  Such  damage  is  a  result  of  the  synergis¬ 
tic  damage  mechanisms  that  are  at  work  in  the 


Table  18-2.  Types  of  Live  Fire  Testing 


LOADING 

FULL-UP 

INERT  A 

FULL  SCALE 

COMPLETE  SYSTEM:  WITH  COMBUSTIBLES 
(E.G.,  BRADLEY  PHASE  II  TESTS,  AIRCRAFT 
“PROOF”  TESTS) 

COMPLETE  SYSTEM:  NO  COMBUSTIBLES  (E.G.,  TESTS  OF 
NEW  ARMOR  ON  ACTUAL  TANKS,  AIRCRAFT  FLIGHT 
CONTROL  TESTS) 

SUB  SCALE 

COMPONENTS,  SUBCOMPONENTS:  WITH 
COMBUSTIBLES  (E.G.,  FUEL  CELL  TESTS, 
BEHIND  ARMOR,  MOCK-UP  AIRCRAFT, 
ENGINE  FIRE  TESTS) 

COMPONENTS,  SUBCOMPONENTS:  STRUCTURES, 
TERMINAL  BALLISTICS,  MUNITIONS  PERFORMANCE, 
BEHIND-ARMOR  TESTS,  WARHEAD  CHARACTERIZATION 
(E.G.,  ARMOR/WARHEAD  INTERACTION  TESTS,  AIRCRAFT 
COMPONENT  STRUCTURAL  TESTS) 

fl  IN  SOME  CASES,  TARGETS  ARE  “SEMI-INERT,”  MEANING  SOME  COMBUSTIBLES  ARE  ON  BOARD,  BUT  NOT  ALL. 
(EXAMPLE:  TESTS  OF  COMPLETE  TANKS  WITH  FUELAND  HYDRAULIC  FLUID,  BUT  DUMMY  AMMUNITION) 

Source:  Live  Fire  Testing:  Evaluating  DoD’s  Program,  General  Accounting  Office,  GAO/PEMD-87-17,  August  1987. 
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“real  world”  and  likely  to  be  found  during  actual 
combat.  LFT  provides  a  way  of  examining  the 
damages  inflicted  not  only  on  materiel  but  also 
on  personnel.  The  crew  casualty  problem  is  an 
important  issue  that  the  LFT  program  is  address¬ 
ing.  The  program  provides  an  opportunity  to  as¬ 
sess  the  effects  of  the  complex  environments  that 
crews  are  likely  to  encounter  in  combat  (e.g.,  fire, 
toxic  fumes,  blunt  injury  shock  and  acoustic 
injuries)  (Reference  36). 


18.2.3  Use  of  Modeling  and  Simulation 
(M&S) 

Survivability  and  lethality  assessments  have  tradi¬ 
tionally  relied  largely  on  the  use  of  modeling  and 
simulation  techniques.  The  LFT  Program  does  not 
replace  the  need  for  such  techniques;  in  fact,  the 
LFT  Guidelines  issued  by  OSD  in  May  1987  (Fig¬ 
ure  18-1)  required  that  no  shots  be  conducted  until 
pre-shot  model  predictions  were  made  concern¬ 
ing  the  expected  damage.  Such  predictions  are 
useful  for  several  reasons.  First,  they  assist  in  the 
test  planning  process.  If  a  model  predicts  that  no 
damage  will  be  inflicted,  test  designers  and 


Figure  18-1.  Live  Fire  Test  and  Evaluation  Planning  Guide 
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planners  should  reexamine  the  selection  of  the  shot 
lines  and/or  reassess  the  accuracy  of  the  threat  rep¬ 
resentation.  Second,  pre-shot  model  predictions  pro¬ 
vide  the  Services  with  the  opportunity  to  validate 
the  accuracy  of  the  models  by  comparing  them  with 
actual  LFT  results.  At  the  same  time,  the  LFT  pro¬ 
gram  reveals  areas  of  damage  that  may  be  absent 
from  existing  models  and  simulations.  Third,  pre¬ 
shot  model  predictions  can  be  used  to  help  conserve 
scarce  target  resources.  For  example,  models  can  be 
used  to  determine  a  sequence  of  shots  that  provides 
for  the  less-damaging  shots  to  be  conducted  first, 
followed  by  the  more-catastrophic  shots  resulting  in 
maximum  target  damage. 

18.2.4  Live  Fire  Test  Best  Practices 

The  Defense  Acquisition  Guidebook  states  that 
plans  for  live  fire  testing  must  be  included  in  the 
TEMP.  Key  points  covered  in  the  LFT  guidelines 
include  the  following: 

•  The  LFT&E  Detailed  T&E  Plan  is  the  basic 
planning  document  used  by  OSD  and  the  Ser¬ 
vices  to  plan,  review  and  approve  LFT&E.  Ser¬ 
vices  will  submit  the  plan  to  the  DOT&E  for 
comment  at  least  30  days  prior  to  test  initiation. 

•  The  LFT&E  plan  must  contain  general  infor¬ 
mation  on  the  system’s  required  performance, 
operational  and  technical  characteristics,  criti¬ 
cal  test  objectives  and  the  evaluation  process. 

•  Each  LFT&E  plan  must  include  testing  of  com¬ 
plete  systems.  A  limited  set  of  live  fire  tests 
may  involve  production  components  configured 
as  a  subsystem  before  full-up  testing. 

•  A  Service  report  must  be  submitted  within  120 
days  of  the  completion  of  the  live  fire  test.  The 
report  must  include  the  firing  results,  test  con¬ 
ditions,  limitations  and  conclusions  and  be  sub¬ 
mitted  in  classified  and  unclassified  form. 

•  Within  45  days  of  receipt  of  the  Service  report, 
a  separate  Live  Fire  Test  Report  (Appendix  3, 


Defense  Acquisition  Guidebook)  will  be  pro¬ 
duced  by  the  DOT&E,  approved  by  the  Secre¬ 
tary  of  Defense,  and  transmitted  to  Congress. 
The  conclusions  of  the  report  will  be  indepen¬ 
dent  of  the  conclusions  of  the  Service  report. 
Reporting  on  LFT&E  may  be  included  in  the 
weapon  system’s  Beyond  Low  Rate  Initial  Pro¬ 
duction  (BLRIP)  Report  completed  by  the 
DOT&E. 

•  The  Congress  shall  have  access  to  all  live  fire 
test  data  and  all  live  fire  test  reports  held  by  or 
produced  by  the  Secretary  of  the  concerned 
Service  or  by  OSD. 

•  The  costs  of  all  live  fire  tests  shall  be  paid  from 
funding  for  the  system  being  tested.  In  some 
instances,  the  Deputy  DOT&E-Live  Fire  may 
elect  to  supplement  such  funds  for  the  acquisi¬ 
tion  of  targets  or  target  simulators,  although  the 
ultimate  responsibility  rests  on  the  concerned 
Service. 

The  Survivability,  Vulnerability  Information  Analy¬ 
sis  Center  (SURVIAC)  is  the  DoD  focal  point  for 
non-nuclear  survivability/vulnerability  data,  infor¬ 
mation,  methodologies,  models,  and  analysis  re¬ 
lating  to  U.S.  and  foreign  aeronautical  and  sur¬ 
face  systems.  Data  on  file  for  conventional  mis¬ 
siles  and  guns  includes  information  on  acquisi¬ 
tion,  detection,  tracking,  launch,  fly-out  and  fuz¬ 
ing  characteristics,  countermeasures  and  counter¬ 
countermeasures,  and  terminal  effects.  Other  weap¬ 
ons  systems  information  includes  physical  and 
functional  characteristics;  design,  performance,  and 
operational  information;  acoustics,  infrared,  opti¬ 
cal,  electro-optical,  and  radar  signature;  combat 
damage  and  repair;  and  system,  subsystem,  and 
component  probability  of  kill  given  a  hit  (pk/h) 
functions.  SURVIAC  is  sponsored  by  the  Joint 
Logistics  Commanders’  Joint  Aircraft  Survivabil¬ 
ity  Program  Office  (http://ias.ics.mill  and  the  Joint 
Technical  Coordinating  Group  on  Munitions  Ef¬ 
fectiveness  (http://jtcg.amsaa.armv.mil/index.htmll. 
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18.3  TESTING  FOR  NUCLEAR 

HARDNESS  AND  SURVIVABILITY 

18.3.1  Background 

Nuclear  survivability  must  be  incorporated  into  the 
design,  acquisition  and  operation  of  all  systems 
that  must  perform  critical  missions  in  a  nuclear 
environment.  Nuclear  survivability  is  achieved 
through  a  combination  of  four  methods:  hardness, 
avoidance,  proliferation  and  reconstitution.  Hard¬ 
ness  allows  a  system  to  physically  withstand  a 
nuclear  attack.  Avoidance  encompasses  measures 
taken  to  avoid  encountering  a  nuclear  environ¬ 
ment.  Proliferation  involves  having  sufficient  sys¬ 
tems  to  compensate  for  probable  losses.  Recon¬ 
stitution  includes  the  actions  taken  to  repair  or  re¬ 
supply  damaged  units  in  time  to  complete  a  mis¬ 
sion  satisfactorily. 

There  is  a  wide  variety  of  possible  effects  from  a 
nuclear  detonation.  They  include:  Electromagnetic 
Pulse  (EMP),  ionizing  radiation,  thermal  radiation, 
blast,  shock,  dust,  debris,  blackout,  and  scintilla¬ 
tion.  Each  weapon  system  is  susceptible  to  some 
but  not  all  of  these  effects.  The  Program  Manager 
(PM)  and  his/her  staff  must  identify  the  effects  that 
may  have  an  impact  on  the  system  under  develop¬ 
ment  and  manage  the  design,  development,  and 
testing  of  the  system  in  a  manner  that  minimizes 
degradation.  The  variety  of  possible  nuclear  effects 
is  described  more  fully  in  the  Nuclear  Survivability 
Handbook  for  Air  Force  OT&E  (Reference  88). 

18.3.2  Assessing  Nuclear  Survivability 
Throughout  the  System  Acquisition 
Cycle 

The  PM  must  ensure  that  nuclear  survivability  is¬ 
sues  are  addressed  throughout  the  system  acquisi¬ 
tion  cycle.  During  assessment  of  concepts,  the 
survivability  requirements  stated  in  the  Service 
requirements  document  should  be  verified,  refined 
or  further  defined.  During  the  system’s  early  de¬ 
sign  stages,  tradeoffs  between  hardness  levels  and 
other  system  characteristics  (such  as  weight, 


decontaminability  and  compatibility)  should  be 
described  quantitatively.  Tradeoffs,  between  hard¬ 
ness,  avoidance,  proliferation  and  reconstitution 
as  a  method  for  achieving  survivability,  should 
also  be  considered  at  this  time.  During  advanced 
engineering  development,  the  system  must  be  ad¬ 
equately  tested  to  confirm  that  hardness  objectives, 
criteria,  requirements  and  specifications  are  met. 
Plans  for  nuclear  hardness  and  survivability  test¬ 
ing  should  be  addressed  in  the  TEMP.  The  ap¬ 
propriate  commands  must  make  provision  for  test 
and  hardness  surveillance  equipment  and  proce¬ 
dures  so  required  hardness  levels  can  be  main¬ 
tained  once  the  system  is  operational. 

During  FRP,  deployment  and  operational  support, 
system  hardness  is  maintained  through  an  active 
hardness  assurance  program.  Such  a  program  en¬ 
sures  that  the  end  product  conforms  to  hardness 
design  specifications  and  that  hardness  aspects  are 
reevaluated  before  any  retrofit  changes  are  made 
to  existing  systems. 

Once  a  system  is  operational,  a  hardness  surveil¬ 
lance  program  may  be  implemented  to  maintain 
system  hardness  and  to  identify  any  further  evalu¬ 
ation,  testing  or  retrofit  changes  required  to  en¬ 
sure  survivability.  A  hardness  surveillance  program 
consists  of  a  set  of  scheduled  tests  and  inspec¬ 
tions  to  ensure  that  a  system’s  designed  hardness 
is  not  degraded  through  operational  use,  logistic 
support,  maintenance  actions,  or  natural  causes. 

18.3.3  Test  Planning 

The  Nuclear  Survivability  Handbook  for  Air  Force 
OT&E  describes  the  following  challenges  associ¬ 
ated  with  nuclear  hardness  and  survivability 
testing: 

(1)  The  magnitude  and  range  of  effects  from  a 
nuclear  burst  are  much  greater  than  those 
from  conventional  explosions  that  may  be 
used  to  simulate  nuclear  bursts.  Nuclear 
detonations  have  effects  not  found  in  con¬ 
ventional  explosions.  The  intense  nuclear 
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Table  18-3.  Nuclear  Hardness  and  Survivability  Assessment  Activities 

CONCEPT ASSESSMENT 

•  PREPARATION  OF  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  THAT  INCLUDES  INITIAL  PLANS  FOR 
NUCLEAR  HARDNESS  AND  SURVIVABILITY  (NH&S)  TESTS 

-  IDENTIFICATION  OF  NH&S  REQUIREMENTS  IN  VERIFIABLE  TERMS 

-  IDENTIFICATION  OF  SPECIAL  NH&S  TEST  FACILITY  REQUIREMENTS  WITH  EMPHASIS  ON  LONG  LEAD 
TIME  ITEMS 

•  DEVELOPMENT  OF  NUCLEAR  CRITERIA 

PROTOTYPETESTING 

•  INCREASE TEST  PLANNING 

•  TEMP  UPDATE 

•  CONDUCT  OF  NH&S  TRADE  STUDIES 

-  NH&S  REQUIREMENTS  VERSUS  OTHER  SYSTEM  REQUIREMENTS 

-  ALTERNATE  METHODS  FOR  ACHIEVING  NH&S 

•  CONDUCT  OF  LIMITED  TESTING 

-  PIECE-PART  HARDNESS  TESTING 

-  DESIGN  CONCEPT  TRADE-OFF  TESTING 

-  TECHNOLOGY  DEMONSTRATION  TESTING 

•  DEVELOPMENT  OF  PERFORMANCE  SPECIFICATIONS  THAT  INCLUDE  QUANTITATIVE  HARDNESS  LEVELS 

ENGINEERING  DEVELOPMENT  MODEL 

•  FIRST  OPPORTUNITY  TO  TEST  PROTOTYPE  HARDWARE 

•  TEMP  UPDATE 

•  DEVELOPMENT  OF  NUCLEAR  HARDNESS  DESIGN  HANDBOOK 

-  PRIOR  TO  PRELIMINARY  DESIGN  REVIEW 

-  USUALLY  PREPARED  BY  NUCLEAR  EFFECTS  SPECIALTY  CONTRACTOR 

•  CONDUCT  OF  TESTING 

-  PRE-CRITICAL  DESIGN  REVIEW  (CDR)  DEVELOPMENT  AND  QUALIFICATION  TEST 

-  DEVELOPMENTALTESTING  ON  NUCLEAR-HARDENED  PIECE  PARTS,  MATERIALS,  CABLING,  AND 
CIRCUITS 

-  NH&S  BOX  AND  SUBSYSTEM  QUALIFICATION  TESTS  (POST-CDR) 

-  ACCEPTANCE  TESTS  TO  VERIFY  HARDWARE  MEETS  SPECIFICATIONS  (POST-CDR,  PRIOR  TO  FIRST 
DELIVERY) 

-  SYSTEM-LEVEL  HARDNESS  ANALYSIS  (USING  BOX  AND  SUBSYSTEM  TEST  RESULTS) 

-  SYSTEM-LEVEL  NH&S  TEST 

PRODUCTION  (LRIP,  FULL  RATE),  DEPLOYMENT  AND  OPERATIONAL  SUPPORT 

•  IMPLEMENTATION  OF  PROGRAM  TO  ENSURE  SYSTEM  RETAINS  ITS  NH&S  PROPERTIES 

•  SCREENING  OF  PRODUCTION  HARDWARE  FOR  HARDNESS 

•  IMPLEMENTATION  BY  USER  OF  PROCEDURES  TO  ENSURE  SYSTEM’S  OPERATION,  LOGISTIC  SUPPORT 
AND  MAINTENANCE  DO  NOT  DEGRADE  HARDNESS  FEATURES 

•  REASSESSMENT  OF  SURVIVABILITY  THROUGHOUT  SYSTEM  LIFE  CYCLE 


18-6 


radiation,  blast,  shock,  thermal,  and  EMP 
fields  are  difficult  to  simulate.  In  addition, 
systems  are  often  tested  at  stress  levels  that 
are  either  lower  than  those  established  by 
the  criteria  or  lower  than  the  level  needed  to 
cause  damage  to  the  system. 

(2)  The  yields  and  configurations  for  under¬ 
ground  testing  are  limited.  It  is  generally  not 
possible  to  test  all  relevant  effects  simulta¬ 
neously  or  to  observe  possibly  important 
synergism  between  effects. 

(3)  System-level  testing  for  nuclear  effects  is 
normally  expensive,  takes  years  to  plan  and 
conduct  and  requires  specialized  expertise. 
Often,  classes  of  tests  conducted  early  in  the 
program  are  not  repeated  later.  Therefore, 
operational  requirements  should  be  folded 
into  these  tests  from  the  start,  often  early  in 
the  acquisition  process.  This  mandates  a 
more-extensive,  combined  DT&E/OT&E 
(Operational  Test  and  Evaluation)  test  pro¬ 
gram  than  normally  found  in  other  types  of 
testing. 

PMs  and  test  managers  must  remain  sensitive  to 
the  ambiguities  involved  in  testing  for  nuclear  sur¬ 
vivability.  For  example,  there  is  no  universal  quan¬ 
titative  measure  of  survivability;  and  statements 
of  survivability  may  lend  themselves  to  a  variety 
of  interpretations.  Moreover,  it  can  be  difficult  to 
combine  system  vulnerability  estimates  for  vari¬ 
ous  nuclear  effects  into  an  assessment  of  overall 
survivability.  As  a  result,  program/test  managers 
must  exercise  caution  when  developing  test  ob¬ 
jectives  and  specifying  measures  of  merit  related 
to  nuclear  survivability. 

18.3.4  Test  Execution 


(OT)  efforts  are  often  combined  because  it  is  not 
possible  to  test  in  an  operational  nuclear  environ¬ 
ment.  The  use  of  an  integrated  DT/OT  program 
requires  early  and  continuous  dialogue  between 
the  two  test  communities  so  each  understands  the 
needs  of  the  other  and  maximum  cooperation  in 
meeting  objectives  is  obtained. 

T&E  techniques  available  to  validate  the  nuclear 
survivability  aspects  of  systems  and  subsystems 
include  underground  nuclear  testing,  environmen¬ 
tal  simulation  (system  level,  subsystem  level  and 
component  level)  and  analytical  simulation.  Table 
18-3  outlines  the  major  activities  relevant  to  the 
assessment  of  nuclear  hardness  and  survivability 
and  the  phases  of  the  acquisition  cycle  in  which 
they  occur. 

18.4  SUMMARY 

The  survivability  and  lethality  aspects  of  certain 
system  must  be  evaluated  through  live  fire  tests. 
These  tests  are  used  to  provide  insights  into  the 
weapon  system’s  ability  to  continue  to  operate/ 
fight  after  being  hit  by  enemy  threat  systems.  It 
provides  a  way  of  examining  the  damages  inflicted 
not  only  on  materiel  but  also  on  personnel.  LFT 
also  provides  an  opportunity  to  assess  the  effects 
of  complex  environments  that  crews  are  likely  to 
encounter  in  combat. 

Nuclear  survivability  must  be  carefully  evaluated 
during  the  system  acquisition  cycle.  Tradeoffs 
between  hardness  levels  and  other  system  charac¬ 
teristics,  such  as  weight,  speed,  range,  cost,  etc., 
must  be  evaluated.  Nuclear  survivability  testing  is 
difficult,  and  the  evaluation  of  test  results  may  lend 
itself  to  a  variety  of  interpretations.  Therefore,  PMs 
must  exercise  caution  when  developing  test  ob¬ 
jectives  related  to  nuclear  survivability. 


For  nuclear  hardness  and  survivability  testing, 
Development  Test  (DT)  and  Operational  Test 
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LOGISTICS 

INFRASTRUCTURE  T&E 


19.1  INTRODUCTION 

In  all  materiel  acquisition  programs,  the  logistics 
support  effort  begins  in  the  capabilities  assessments 
of  mission  needs,  leads  to  the  development  of  the 
Initial  Capabilities  Document  (ICD)  before  pro¬ 
gram  initiation,  continues  throughout  the  acquisi¬ 
tion  cycle  and  extends  past  the  deployment  phase. 
Logistics  support  system  testing  must,  therefore, 
extend  over  the  entire  acquisition  cycle  of  the  sys¬ 
tem  and  be  carefully  planned  and  executed  to  en¬ 
sure  the  readiness  and  supportability  of  the  sys¬ 
tem.  Discussion  of  Performance  Based  Logistics 
(PBL)  can  be  found  at  the  DAU  Website  (http:// 
www.dau.miD  Acquisition  Knowledge  Sharing 
Site  (AKSS)  for  the  Logistics  Community  of  Prac¬ 
tice  (CoP).  This  chapter  covers  the  development 
of  logistics  support  test  requirements  and  the  con¬ 
duct  of  supportability  assessments  to  ensure  that 
readiness  and  supportability  objectives  are  identi¬ 
fied  and  achieved.  The  importance  of  the  logis¬ 
tics  manager’s  participation  in  the  Test  and  Evalu¬ 
ation  Master  Plan  (TEMP)  development  process 
should  be  stressed.  The  logistics  manager  must 
ensure  the  logistics  system  Test  and  Evaluation 
(T&E)  objectives  are  considered  and  that  adequate 
resources  are  available  for  logistics  support  system 
T&E. 

Logistic  system  support  planning  is  a  disciplined, 
unified  and  iterative  approach  to  the  management 
and  technical  activities  necessary  to  integrate  sup¬ 
port  considerations  into  system  and  equipment 
design;  develop  support  requirements  that  are  re¬ 
lated  consistently  to  readiness  objectives,  design, 
and  each  other;  acquire  the  required  support;  and 
provide  the  required  support  during  deployment 
and  operational  support  at  affordable  cost. 


Logistic  support  systems  are  usually  categorized 
into  10  specific  components,  or  elements: 

(1)  Maintenance  planning 

(2)  Manpower  and  personnel 

(3)  Supply  support 

(4)  Support  equipment 

(5)  Technical  data 

(6)  Training  and  training  support 

(7)  Computer  resources  support 

(8)  Facilities 

(9)  Packaging,  handling,  storage,  and  transportation 

(10)  Design  interface. 

19.2  PLANNING  FOR  LOGISTIC 
SUPPORT  SYSTEM  T&E 

19.2.1  Objectives  of  Logistic  System  T &E 

The  main  objective  of  logistic  system  T&E  is  to 
verify  that  the  logistic  support  being  developed 
for  the  materiel  system  is  capable  of  meeting  the 
required  objectives  for  peacetime  and  wartime  em¬ 
ployment.  This  T&E  consists  of  the  usual  Devel¬ 
opment  Test  and  Evaluation  (DT&E)  and  Opera¬ 
tional  Test  and  Evaluation  (OT&E)  but  also  in¬ 
cludes  post  deployment  supportability  assess¬ 
ments.  The  formal  DT&E  and  OT&E  begin  with 
sub-component  assembly  and  prototype  testing, 
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Figure  19-1.  Logistics  Supportability  Objectives  in  theT&E  Program 


and  continuing  throughout  advanced  engineering 
development,  production,  deployment,  and  opera¬ 
tional  support.  Figure  19-1  describes  the  specific 
Development  Test  (DT),  Operational  Test  (OT) 
and  supportability  assessment  objectives  for  each 
acquisition  phase.  “The  PM  shall  apply  Human 
Systems  Integration  (HSI)  to  optimize  total  sys¬ 
tem  performance  (hardware,  software,  and  hu¬ 
man),  operational  effectiveness,  and  suitability,  sur¬ 
vivability,  safety,  and  affordability.”  (Department 
of  Defense  Directive  (DoDD)  5000.1,  The  De¬ 
fense  Acquisition  System,  12  May  2003.) 

19.2.2  Planning  for  Logistic  Support  System 
T&E 

19.2.2.1  Logistic  Support  Planning 

“The  PM  shall  have  a  comprehensive  plan  for  HSI 
in  place  early  in  the  acquisition  process  to  opti¬ 
mize  total  system  performance,  minimize  Total 
Ownership  Costs  [TOCs],  and  ensure  that  the  sys¬ 
tem  is  built  to  accommodate  the  characteristics  of 
the  user  population  that  will  operate,  maintain,  and 
support  the  system.”  HSI  includes  design  for:  hu¬ 
man  factors  engineering;  personnel;  habitability; 
manpower;  training;  Environment,  Safety,  and 
Occupational  Health  (ESOH);  and  survivability. 
(DoD  Instruction  (DoDI)  5000.2,  Operation  of  the 
Defense  Acquisition  System,  12  May  2003.)  The 
logistic  support  manager  for  a  materiel  acquisi¬ 
tion  system  is  responsible  for  developing  the  lo¬ 
gistic  support  planning  which  documents  planning 
for  and  implementing  the  support  of  the  fielded 
system.  It  is  initially  prepared  during  preparation 
for  program  initiation,  and  progressively  developed 
in  more  detail  as  the  system  moves  through  the 
acquisition  phases.  Identification  of  the  specific 
logistic  support  test  issues  related  to  the  individual 
logistics  elements  and  the  overall  system  support 
and  readiness  objectives  are  included. 

The  logistic  support  manager  is  assisted  through¬ 
out  the  system’s  development  by  a  Logistics  Sup¬ 
port  Integrated  Product  Team  (IPT)  which  is 
formed  early  in  the  acquisition  cycle.  The 


Logistics  Support  IPT  is  a  coordination/advisory 
group  composed  of  personnel  from  the  Program 
Management  Office  (PMO),  the  using  command 
and  other  commands  concerned  with  acquisition 
activities  such  as  logistics,  testing,  and  training. 

19.2.2.2  Supportability  Assessment  Planning 

Based  upon  suitability  objectives,  the  logistic  sup¬ 
port  manager,  in  conjunction  with  the  system’s  Test 
Manager  (TM),  develops  suitability  assessment 
planning.  This  planning  identifies  the  testing  ap¬ 
proach  and  the  evaluation  criteria  that  will  be  used 
to  assess  the  supportability-related  design 
requirements  (e.g.,  Reliability  and  Maintainability 
(R&M))  and  adequacy  of  the  planned  logistic  sup¬ 
port  resources  for  the  materiel  system.  Develop¬ 
ment  of  the  suitability  T&E  planning  begins  dur¬ 
ing  concept  refinement;  the  planning  is  then  up¬ 
dated  and  refined  in  each  successive  acquisition 
phase.  The  logistic  support  manager  may  apply 
the  best  practices  of  logistic  support  analysis  as 
described  in  MIL-HDBK-1388-1A. 

The  T&E  strategy  is  formulated,  T&E  program 
objectives  and  criteria  are  established,  and  required 
test  resources  are  identified.  The  logistic  support 
manager  ensures  that  T&E  strategy  is  based  upon 
quantified  supportability  requirements  and  ad¬ 
dresses  supportability  issues  including  those  with 
a  high  degree  of  associated  risk.  Also,  the  logistic 
support  manager  ensures  that  the  necessary  quan¬ 
tities  and  types  of  data  will  be  collected  during 
system  development  and  after  deployment  of  the 
system  to  validate  the  various  T&E  objectives.  The 
T&E  objectives  and  criteria  must  provide  a  basis 
that  ensures  critical  supportability  issues  and  re¬ 
quirements  are  resolved  or  achieved  within 
acceptable  confidence  levels. 

19.2.2.3  Test  and  Evaluation  Master  Plan 
(TEMP) 

The  Program  Manager  (PM)  must  include  suit¬ 
ability  T&E  information  in  the  TEMP  as  speci¬ 
fied  in  the  Defense  Acquisition  Guidebook.  The 
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input,  derived  from  logistic  supportability  planning 
with  the  assistance  of  the  logistic  support  manager 
and  the  tester,  includes  descriptions  of  required 
operational  suitability,  specific  plans  for  testing 
logistics  supportability,  and  required  testing  re¬ 
sources.  It  is  of  critical  importance  that  all  key 
test  resources  required  for  ILS  testing  (DT,  OT, 
and  post  deployment  supportability)  be  identified 
in  the  TEMP  because  the  TEMP  provides  a  long- 
range  alert  upon  which  test  resources  are  budgeted 
and  obtained  for  testing. 

19.2.3  Planning  Guidelines  for  Logistic 
Support  System  T&E 

The  following  guidelines  were  selected  from  those 
listed  in  the  Defense  Systems  Management 
College’s  Acquisition  Logistic  Guide  (December 
1997): 

(1)  Develop  a  test  strategy  for  each  logistics  sup- 
port-related  objective.  Ensure  that  OT&E 
planning  encompasses  all  logistic  support  el¬ 
ements.  The  general  objectives  shown  in 
Figure  19-1  must  be  translated  into  detailed 
quantitative  and  qualitative  requirements  for 
each  acquisition  phase  and  each  T&E 
program. 

(2)  Incorporate  logistic  support  testing  require¬ 
ments  (where  feasible)  into  the  formal 
DT&E/OT&E  plans. 

(3)  Identify  logistic  support  T&E  that  will  be 
performed  outside  of  the  normal  DT&E  and 
OT&E.  Include  subsystems  that  require  off- 
system  evaluation. 

(4)  Identify  all  required  resources,  including  test 
articles  and  logistic  support  items  for  formal 
DT/OT  and  separate  logistic  support  system 
testing  (participate  with  test  planner). 

(5)  Ensure  establishment  of  an  operationally  re¬ 
alistic  test  environment,  to  include  person¬ 
nel  representatives  of  those  who  will  even¬ 


tually  operate  and  maintain  the  fielded 
system.  These  personnel  should  be  trained 
for  the  test  using  prototypes  of  the  actual 
training  courses  and  devices.  They  should 
be  supplied  with  drafts  of  all  technical  manu¬ 
als  and  documentation  that  will  be  used  with 
the  fielded  system. 

(6)  Ensure  planned  OT&E  will  provide  suffi¬ 
cient  data  on  high-cost  and  high-maintenance 
burden  items  (e.g.,  for  high-cost  critical 
spares,  early  test  results  can  be  used  to 
reevaluate  selection). 

(7)  Participate  early  and  effectively  in  the  TEMP 
development  process  to  ensure  the  TEMP 
includes  critical  logistic  T&E  designated  test 
funds  from  program  and  budget  documents. 

(8)  Identify  the  planned  utilization  of  all  data 
collected  during  the  assessments  to  avoid 
mismatching  of  data  collection  and  informa¬ 
tion  requirements. 

Additional  guidance  may  be  found  in  the  Logis¬ 
tics  Test  and  Evaluation  Handbook  developed  by 
the  412  Logistics  Group,  Edwards  Air  Force 
Base,  California. 

19.3  CONDUCTING  LOGISTIC  SUPPORT 
SYSTEM  T&E 

19.3.1  The  Process 

The  purposes  of  logistic  support  system  T&E  are 
to  measure  the  supportability  of  a  developing  sys¬ 
tem  throughout  the  acquisition  process,  to  iden¬ 
tify  supportability  deficiencies  and  potential  cor¬ 
rections/improvements  as  test  data  becomes  avail¬ 
able,  and  to  assess  the  operational  suitability  of 
the  planned  support  system.  It  also  evaluates  the 
system’s  ability  to  achieve  planned  readiness  ob¬ 
jectives  for  the  system/equipment  being  developed. 
Specific  logistic  support  system  T&E  tasks  (guid¬ 
ance  in  MIL-HDBK- 1 388- 1 A)  include: 
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•  Analysis  of  test  results  to  verify  achievement 
of  specified  supportability  requirements; 

•  Determination  of  improvements  in  supportabil¬ 
ity  and  supportability-related  design  parameters 
needed  for  the  system  to  meet  established  goals 
and  thresholds; 

•  Identification  of  areas  where  established  goals 
and  thresholds  have  not  been  demonstrated 
within  acceptable  confidence  levels; 

•  Development  of  corrections  for  identified  sup- 
portability  problems  such  as  modifications  to 
hardware,  software,  support  plans,  logistic  sup¬ 
port  resources  or  operational  tactics; 

•  Projection  of  changes  in  costs,  readiness  and 
logistic  support  resources  due  to  implementa¬ 
tion  of  corrections; 

•  Analysis  of  supportability  data  from  the  de¬ 
ployed  system  to  verify  achievement  of  the 
established  goals  and  thresholds  and  where 
operational  results  deviate  from  projections, 
determination  of  the  causes  and  corrective 
actions. 

Logistics  support  system  T&E  may  consist  of  a 
series  of  logistics  demonstrations  and  assessments 
that  are  usually  conducted  as  part  of  system  per¬ 
formance  tests.  Special  end-item  equipment  tests 
are  rarely  conducted  solely  for  logistic  parameter 
evaluation. 

19.3.2  Reliability  and  Maintainability  (R&M) 

System  availability  is  generally  considered  to  be 
composed  of  two  major  system  characteristics  — 
Reliability  and  Maintainability  (R&M).  Reliabil¬ 
ity,  Availability,  and  Maintainability  (RAM)  re¬ 
quirements  should  be  based  on  operational  re¬ 
quirements  and  life-cycle  cost  considerations; 
stated  in  quantifiable,  operational  terms;  measur¬ 
able  during  developmental  and  operational  test  and 
evaluation;  and  defined  for  all  elements  of  the 


system,  including  support  and  training  equipment. 
Relevant  definitions  from  the  DAU  Glossary: 

Reliability  (R)  is  the  ability  of  a  system  and 
its  parts  to  perform  its  mission  without  fail¬ 
ure,  degradation,  or  demand  on  the  support 
system.  A  common  MOE  has  been  Mean 
Time  Between  Failure  (MTBF). 

Maintainability  (M)  is  the  ability  of  an  item 
to  be  retained  in  or  restored  to  specified 
condition  when  maintenance  is  performed 
by  personnel  having  specific  skill  levels, 
using  prescribed  procedures  and  resources, 
at  each  prescribed  level  of  maintenance 
and  repair.  A  common  MOE  has  been 
Mean  Time  To  Repair  (MTTR). 

Operational  Reliability  and  Maintainability 
Value  is  any  measure  of  reliability  or  main¬ 
tainability  that  includes  the  combined  effects 
of  item  design,  quality,  installation,  environ¬ 
ment,  operation,  maintenance,  and  repair. 

The  R  and  M  program  objectives  are  to  be  de¬ 
fined  as  system  parameters  early  in  the  develop¬ 
ment  process.  They  will  be  used  as  evaluation 
criteria  throughout  the  design,  development  and 
production  processes.  R&M  objectives  should  be 
translated  into  quantifiable  contractual  terms  and 
allocated  through  the  system  design  hierarchy.  An 
understanding  of  how  this  allocation  affects  test¬ 
ing  operating  characteristics  below  system  level 
can  be  found  in  DoD  3235. 1-H,  T&E  of  System 
Reliability,  Availability  and  Maintainability.  This 
is  especially  important  to  testing  organizations 
expected  to  make  early  predictions  of  system  per¬ 
formance.  Guidance  on  testing  reliability  may  also 
be  found  in  MIL-HDBK-78I,  Reliability  Testing 
for  Engineering  Development,  Qualification,  and 
Production. 

Reliability,  Availability,  and  Maintainabil¬ 
ity  (also  know  as  RAM),  are  requirements 
imposed  on  acquisition  systems  to  insure 
they  are  operationally  ready  for  use  when 
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needed,  will  successfully  perform  assigned 
functions,  and  can  be  economically  oper¬ 
ated  and  maintained  within  the  scope  of 
logistics  concepts  and  policies.  RAM  de¬ 
velopment  programs  are  applicable  to  ma¬ 
teriel  systems,  test  measurement  and  diag¬ 
nostic  equipment,  training  devices,  and  fa¬ 
cilities  supporting  weapon  systems. 

Availability  is  a  measure  of  the  degree  to 
which  an  item  is  in  an  operable  state  and 
can  be  committed  at  the  start  of  a  mission 
when  the  mission  is  called  for  at  an  un¬ 
known  (random)  point  in  time. 

Availability  is  commonly  uses  different  MOE  based 
on  the  maturity  of  the  weapon  system  technology. 
Achieved  availability  (Aa)  is  measured  for  the  early 
prototype  and  EDM  systems  developed  by  the  sys¬ 
tem  contractor  when  the  system  is  not  operating  in 
its  normal  environment.  Inherent  Availability  (Ai) 
is  measured  with  respect  only  to  operating  time  and 
corrective  maintenance,  ignoring  standby/delay  time 
and  mean  logistics  delay  time.  It  may  be  measured 
during  DT&E  or  when  testing  in  the  contractor’s 
facility  under  controlled  conditions.  Operational 
Availability  (Ao),  measured  for  mature  systems  that 
have  been  deployed  in  realistic  operational  envi¬ 
ronments,  is  the  degree  to  which  a  piece  of  equip¬ 
ment  works  properly  when  it  is  required  for  a  mis¬ 
sion.  It  is  calculated  by  dividing  uptime  by  uptime 
plus  all  downtime.  The  calculation  is  sensitive  to 
low  utilization  rates  for  equipment.  Ao  is  the  quan¬ 
titative  link  between  readiness  objectives  and  sup- 
portability.  (DoD  3235. 1-H) 

19.3.2.1  Reliability 

Guidelines  for  reliability  evaluation  are  to  be  pub¬ 
lished  in  a  non-government  standard  to  replace 
MIL-STD-785: 

Environmental  Stress  Screening  (ESS) 

is  a  test,  or  series  of  tests  during  engineer¬ 
ing  development,  specifically  designed  to 
identify  weak  parts  or  manufacturing 


defects.  Test  conditions  should  stimulate 
failures  typical  of  early  field  service  rather 
than  provide  an  operational  life  profile. 

Reliability  Development/Growth  Testing  (RDT/ 
RGT)  is  a  systematic  engineering  process  of  Test- 
Analyze-Fix-Test  (TAFT)  where  equipment  is 
tested  under  actual,  simulated,  or  accelerated  en¬ 
vironments.  It  is  an  iterative  methodology  intended 
to  rapidly  and  steadily  improve  reliability.  Test  ar¬ 
ticles  are  usually  subjected  to  ESS  prior  to  begin¬ 
ning  RDT/RGT  to  eliminate  those  with  manufac¬ 
turing  defects. 

Reliability  Qualification  Test  (RQT)  is  to  verify 
that  threshold  reliability  requirements  have  been 
met  before  items  are  committed  to  production.  A 
statistical  test  plan  is  used  to  predefine  criteria 
which  will  limit  government  risk.  Test  conditions 
must  be  operationally  realistic. 

Production  Reliability  Acceptance  Test  (PRAT) 

is  intended  to  simulate  in-service  use  of  the  deliv¬ 
ered  item  or  production  lot.  “Because  it  must  pro¬ 
vide  a  basis  for  determining  contractual  compli¬ 
ance,  and  because  it  applies  to  the  items  actually 
delivered  to  operational  forces,  PRAT  must  be 
independent  of  the  supplier  if  at  all  possible.” 
PRAT  may  require  expensive  test  facilities,  so  100 
percent  sampling  is  not  recommended. 

19.3.2.2  Maintainability 

Maintainability  design  factors  and  test/demonstra¬ 
tion  requirements  used  to  evaluate  maintainability 
characteristics  must  be  based  on  program  objec¬ 
tives  and  thresholds.  Areas  for  evaluation  might 
include  (DoD  3235. 1-H): 

Accessibility:  Assess  how  easily  the  item 
can  be  repaired  or  adjusted. 

Visibility:  Assess  the  ability/need  to  see 
the  item  being  repaired. 
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Testability:  Assess  ability  to  detect  and 
isolate  system  faults  to  the  faulty  replace¬ 
able  assembly  level. 

Complexity:  Assess  the  impact  of  the 
number,  location  and  characteristic 
(standard  or  special  purpose)  on  system 
maintenance. 

Interchangeability:  Assess  the  level  of 
difficulty  encountered  when  failed  or  mal¬ 
functioning  parts  are  removed  or  replaced 
with  an  identical  part  not  requiring 
recalibration. 

A  true  assessment  of  system  maintainability  gener¬ 
ally  must  be  developed  at  the  system  level  under 
operating  conditions  and  using  production  configu¬ 
ration  hardware.  Therefore  a  maintainability  dem¬ 
onstration  (guidelines  in  MEL-HDBK-470)  should 
be  conducted  prior  to  Full  Rate  Production  (FRP). 

19.3.3  T&E  of  System  Support  Package 

The  T&E  of  the  support  for  a  materiel  system  re¬ 
quires  a  system  support  package  consisting  of 
spares,  support  equipment,  technical  documents 
and  publications,  representative  personnel,  any 
peculiar  support  requirements  and  the  test  article 
itself,  in  short,  all  of  the  items  that  would  eventu¬ 
ally  be  required  when  the  system  is  operational. 
This  complete  support  package  must  be  at  the  test 
site  before  the  test  is  scheduled  to  begin.  Delays 
in  the  availability  of  certain  support  items  could 
prevent  the  test  from  proceeding  on  schedule.  This 
could  be  costly  due  to  on-site  support  personnel 
on  hold  or  tightly  scheduled  system  ranges  and 
expensive  test  resources  not  being  properly  uti¬ 
lized.  Also,  it  could  result  in  the  test  proceeding 
without  conducting  the  complete  evaluation  of  the 
support  system.  The  logistic  support  test  planner 
must  ensure  that  the  required  personnel  are  trained 
and  available,  the  test  facility  scheduling  is  flex¬ 
ible  enough  to  permit  normal  delays,  and  the  test 
support  package  is  on  site  on  time. 


19.3.4  Data  Collection  and  Analysis 

The  logistic  support  manager  must  coordinate  with 
the  testers  to  ensure  that  the  methods  used  for 
collection,  storage,  and  extraction  of  logistic  sup¬ 
port  system  T&E  data  are  compatible  with  those 
used  in  testing  the  materiel  system.  As  with  any 
testing,  the  test  planning  must  ensure  that  all  re¬ 
quired  data  is  identified;  it  is  sufficient  to  evaluate 
a  system’s  readiness  and  supportability;  and  plans 
are  made  for  a  data  management  system  that  is 
capable  of  the  data  classification,  storage,  retrieval 
and  reduction  necessary  for  statistical  analysis. 
Large  statistical  sample  sizes  may  require  a  com¬ 
mon  database  that  integrates  contractor,  DT&E  and 
OT&E  data  so  required  performance  parameters 
can  be  demonstrated. 

19.3.5  Use  of  Logistic  Support  System  Test 
Results 

The  emphasis  on  the  use  of  the  results  of  testing 
changes  as  the  program  moves  from  the  concept 
assessments  to  post  deployment.  During  early 
phases  of  a  program,  the  evaluation  results  are 
used  primarily  to  verify  analysis  and  develop  fu¬ 
ture  projections.  As  the  program  moves  into  ad¬ 
vanced  engineering  development  and  hardware 
becomes  available,  the  evaluation  addresses  de¬ 
sign,  particularly  the  R&M  aspects;  training  pro¬ 
grams;  support  equipment  adequacy;  personnel 
skills  and  availability;  and  technical  publications. 

The  logistic  support  system  manager  must  make 
the  PM  aware  of  the  impact  on  the  program  of 
logistical  shortcomings  that  are  identified  during 
the  T&E  process.  The  PM,  in  turn,  must  ensure 
that  the  solutions  to  any  shortcomings  are  identi¬ 
fied  and  reflected  in  the  revised  specifications  and 
that  the  revised  test  requirements  are  included  in 
the  updated  TEMP  as  the  program  proceeds 
through  the  various  acquisition  stages. 
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19.4  LIMITATIONS  TO  LOGISTIC 
SUPPORT  SYSTEM  T&E 

Concurrent  testing  or  tests  that  have  accelerated 
schedules  frequently  do  not  have  sufficient  test 
articles,  equipment  or  hardware  to  achieve  statis¬ 
tical  confidence  in  the  testing  conducted.  An 
acquisition  strategy  may  stipulate  that  support  re¬ 
sources  such  as  operator  and  maintenance  manu¬ 
als,  tools,  support  equipment,  training  devices,  etc. 
for  major  weapon  system  components  would  not 
be  procured  before  the  weapons  system/ 
component  hardware  and  software  design  stabi¬ 
lizes.  This  shortage  of  equipment  is  often  the  rea¬ 
son  that  shelf-life  and  service-life  testing  is  incom¬ 
plete,  leaving  the  logistic  support  system  evalua¬ 
tor  with  insufficient  data  to  predict  future  perfor¬ 
mance  of  the  test  item.  Some  evaluations  must 
measure  performance  against  a  point  on  the 
parameter’s  growth  curve.  The  logistic  support 
system  testing  will  continue  post  production  to 
obtain  required  sample  sizes  for  verifying  perfor¬ 
mance  criteria.  Many  aspects  of  the  logistic  sup¬ 
port  system  may  not  be  available  for  Initial  Op¬ 
erational  Test  and  Evaluation  (IOT&E)  and  be¬ 
come  testing  limitations.  The  PMO  must  develop 
enough  logistic  support  to  ensure  the  user  can 
maintain  the  system  during  IOT&E  without 


requiring  system  contractor  involvement  (legislated 
constraints).  Any  logistics  support  system  limita¬ 
tions  upon  IOT&E  will  likely  be  evaluated  during 
Follow-on  Operational  Test  and  Evaluation 
(FOT&E). 

19.5  SUMMARY 

T&E  are  the  logisticians’  tools  for  measuring  the 
ability  of  the  planned  support  system  to  fulfill  the 
materiel  system’s  readiness  and  supportability  ob¬ 
jectives.  The  effectiveness  of  logistic  support  sys¬ 
tem  T&E  is  based  upon  the  completeness  and 
timeliness  of  the  planning  effort. 

The  logistics  support  system  T&E  requirements 
must  be  an  integral  part  of  the  TEMP  to  ensure 
budgeting  and  scheduling  of  required  test  re¬ 
sources.  Data  requirements  must  be  completely 
identified,  with  adequate  plans  made  for  collec¬ 
tion,  storage,  retrieval  and  reduction  of  test  data. 
At  the  Full  Rate  Production  Decision  Review 
(FRPDR),  decision  makers  can  expect  that  some 
logistics  system  performance  parameters  will  not 
have  finished  testing  because  of  the  large  sample 
sizes  required  for  high  confidence  levels  in 
statistical  analysis. 
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EC/C4ISR  TEST 
AND  EVALUATION 


20.1  INTRODUCTION 

Testing  of  Electronic  Combat  (EC)  and  Command, 
Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C*ISR)  systems 
pose  unique  problems  for  the  tester  because  of  the 
difficulty  in  measuring  their  operational  perfor¬ 
mance.  Compatibility,  Interoperability,  and  Integra¬ 
tion  (CII)  are  key  performance  areas  for  these  sys¬ 
tems.  Special  testing  techniques  and  facilities  are 
normally  required  in  EC  and  C4ISR  testing.  This 
chapter  discusses  the  problems  associated  with  EC 
and  C4ISR  testing  and  presents  methodologies  the 
tester  can  consider  using  to  overcome  the  problems. 

20.2  TESTING  EC  SYSTEMS 

20.2.1  Special  Consideration  When  Testing 
EC  Systems 

Electronic  combat  systems  operate  across  the  elec¬ 
tromagnetic  spectrum,  performing  offensive  and 
defensive  support  roles.  Configurations  vary  from 
subsystem  components  to  full-up  independent  sys¬ 
tems.  The  EC  systems  are  used  to  increase  sur¬ 
vivability,  degrade  enemy  capability  and  contrib¬ 
ute  to  the  overall  success  of  the  combat  mission. 
Decision  makers  want  to  know  the  incremental 
contribution  to  total  force  effectiveness  made  by 
a  new  EC  system  when  measured  in  a  force-on- 
force  engagement.  However,  the  contractual  speci¬ 
fications  for  EC  systems  are  usually  stated  in  terms 
of  engineering  parameters  such  as  effective  radi¬ 
ated  power,  reduction  in  communications  intelli¬ 
gibility  and  jamming-to-signal  ratio.  These  mea¬ 
sures  are  of  little  use  by  themselves  in  assessing 
contribution  to  mission  success.  The  decision 
makers  require  that  testing  be  conducted  under 


realistic  operational  conditions;  but  the  major  field 
test  ranges,  such  as  the  shoreline  at  Eglin  Air 
Force  Base  (AFB),  Florida,  or  the  desert  at  Nellis 
AFB,  Nevada,  cannot  provide  the  signal  density 
or  realism  of  threats  that  would  be  presented  by 
regional  conflicts  in  central  Europe.  In  field  test¬ 
ing,  the  tester  can  achieve  one-on-one  or,  at  best, 
few-on-few  testing  conditions.  To  do  this  the  tester 
needs  a  methodology  that  will  permit  extrapola¬ 
tion  of  engineering  measurements  and  one-on-one 
test  events  to  create  more  operationally  meaning¬ 
ful  measures  of  mission  success  in  a  force-on-force 
context,  usually  under  simulated  conditions. 

20.2.2  Integrated  Test  Approach 

An  integrated  approach  to  EC  testing  using  a  com¬ 
bination  of  large-scale  models,  computer  simula¬ 
tions,  hybrid  man-in-the-loop  simulators  and  field 
test  ranges  is  a  solution  for  the  EC  tester.  No  tool 
by  itself  is  adequate  to  provide  a  comprehensive 
evaluation.  Simulation,  both  digital  and  hybrid,  can 
provide  a  means  for  efficient  test  execution.  Com¬ 
puter  models  can  be  used  to  simulate  many  dif¬ 
ferent  test  cases  to  aid  the  tester  in  assessing  the 
critical  test  issues  (i.e.,  sensitivity  analysis)  and  pro¬ 
duce  a  comprehensive  set  of  predicted  results.  As 
digital  simulation  models  are  validated  with  em¬ 
pirical  data  from  testing,  they  can  be  used  to  evalu¬ 
ate  the  system  under  test  in  a  more  dense  and  com¬ 
plex  threat  environment  and  at  expected  wartime 
levels.  In  addition,  the  field  test  results  are  used  to 
validate  the  model;  and  the  model  is  used  to  vali¬ 
date  the  field  tests,  thus  lending  more  credibility 
to  both  results.  Hybrid  man-in-the-loop  simulators, 
such  as  the  Real-Time  Electromagnetic  Digitally 
Controlled  Analyzer  and  Processor  (REDCAP) 
and  the  Air  Force  Electronic  Warfare  Evaluation 
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Simulator  (AFEWES)  can  provide  a  capability  to 
test  against  new  threats.  Hybrid  simulators  cost 
less  and  are  safer  than  field  testing.  The  field  test 
ranges  are  used  when  a  wider  range  of  actions 
and  reactions  by  aircraft  and  ground  threat  sys¬ 
tem  operations  is  required. 

Where  one  tool  is  weak,  another  may  be  strong. 
By  using  all  the  tools,  an  EC  tester  can  do  a  com¬ 
plete  job  of  testing.  An  example  of  an  integrated 
methodology  is  shown  in  Figure  20-1.  The  EC 
integrated  testing  can  be  summarized  as: 

(1)  Initial  modeling  phase  for  sensitivity  analysis 
and  test  planning, 


(2)  Active  test  phases  at  hybrid  laboratory 
simulator  and  field  range  facilities, 

(3)  Test  data  reduction  and  analysis, 

(4)  Post-test  modeling  phase  repeating  the  first 
step  using  test  data  for  extrapolation, 

(5)  Force  effectiveness  modeling  and  analysis 
phase  to  determine  the  incremental  con¬ 
tribution  of  the  new  system  to  total  force 
effectiveness. 

Another  alternative  is  the  electronic  combat  test 
process  proposed  in  the  Air  Force  Electronic 
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Figure  20-1.  Integrated  EC  Testing  Approach 
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Combat  Development  Test  Process  Guide,  May 
1991,  issued  by  what  is  now  the  Air  Staff  T&E 
Element,  AF/TE.  The  six  step  process  described 
here  is  graphically  represented  by  Figure  20-2: 

(1)  Deriving  test  requirements, 

(2)  Conducting  pretest  analysis  to  predict  EC 
system  performance, 

(3)  Conducting  test  sequences  under  pro¬ 
gressively  more  rigorous  ground-  and  flight- 
test  conditions, 

(4)  Processing  test  data, 

(5)  Conducting  post-test  analysis  and  evaluation 
of  operational  effectiveness  and  suitability, 


(6)  Feeding  results  back  to  the  system;  devel¬ 
opment  employment  process. 

As  can  be  seen  from  Figure  20-3,  assuming  a  lim¬ 
ited  budget  and  field  test  being  the  most  expen¬ 
sive  per  number  of  trials,  the  cost  of  test  trials  forces 
the  developer  and  tester  to  make  tradeoffs  to  ob¬ 
tain  the  necessary  test  data.  Many  more  iterations 
of  a  computer  simulation  can  be  run  for  the  cost 
of  an  open-air  test. 

20.3  TESTING  OF  C4ISR  SYSTEMS 

20.3.1  Special  Considerations  When  Testing 
C4ISR  Systems 

The  purpose  of  a  C4ISR  system  is  to  provide  a 
commander  with  timely  and  relevant  information 


Figure  20-2.  EC  Test  Process  Concept 
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Figure  20-3.  EC  Test  Resource  Categories 


to  support  sound  war  fighting  decision-making. 
A  variety  of  problems  face  the  C4ISR  system 
tester.  However,  in  evaluating  command  effec¬ 
tiveness,  it  is  difficult  to  separate  the  contribu¬ 
tion  made  by  the  C4ISR  system  from  the  contri¬ 
bution  made  by  the  commander’s  innate,  cogni¬ 
tive  processes.  To  assess  a  C4ISR  system  in  its 
operational  environment,  it  must  be  connected  to 
the  other  systems  with  which  it  would  normally 
operate,  making  traceability  of  test  results  diffi¬ 
cult.  Additionally,  modern  C4ISR  systems  are 
software  intensive  and  highly  interactive,  with 
complex  man-machine  interfaces.  Measuring 
C4ISR  system  effectiveness  thus  requires  the 
tester  to  use  properly  trained  user  troops  during 
the  test  and  to  closely  monitor  software  Test  and 
Evaluation  (T&E).  The  C4ISR  systems  of  defense 
agencies  and  the  Services  (Army,  Navy,  Air 
Force  and  Marines)  are  expected  to  interoperate 
with  each  other  and  with  those  of  the  North  At¬ 
lantic  Treaty  Organization  (NATO)  forces;  hence, 


the  tester  must  also  ensure  inter-Service  and 
NATO  CD. 

20.3.2  C4I  Test  Facilities 

Testing  of  Command,  Control,  Communications, 
Computer  and  Intelligence  (C4I)  systems  will 
have  to  rely  more  on  the  use  of  computer  simu¬ 
lations  and  C4I  test-beds  to  assess  their  overall 
effectiveness.  The  Defense  Information  Systems 
Agency  (DISA)  is  responsible  for  ensuring  in¬ 
teroperability  among  all  U.S.  tactical  C4I  systems 
that  would  be  used  in  joint  or  combined  opera¬ 
tions,  directs  the  Joint  Interoperability  Test  Com¬ 
mand  (JITC)  at  Ft.  Huachuca,  Arizona.  The  JITC 
is  a  test-bed  for  C4I  systems  CII. 
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20.4  TRENDS  IN  TESTING  C4I  SYSTEMS 

20.4.1  Evolutionary  Acquisition  of  C4I 
Systems 

Evolutionary  Acquisition  (EA)  is  a  strategy  de¬ 
signed  to  provide  an  early,  useful  capability  even 
though  detailed  overall  system  requirements  can¬ 
not  be  fully  defined  at  the  program’s  inception.  The 
EA  strategy  contributes  to  a  reduction  in  the  risks 
involved  in  system  acquisition,  since  the  system  is 
developed  and  tested  in  manageable  increments.  The 
C4I  systems  are  likely  candidates  for  EA  because 
they  are  characterized  by  system  requirements  that 
are  difficult  to  quantify  or  even  articulate  and  that 
are  expected  to  change  as  a  function  of  scenario, 
mission,  theater,  threat,  and  emerging  technology. 
Therefore,  the  risk  associated  with  developing  these 
systems  can  be  very  great. 

Studies  by  the  Defense  Systems  Management 
College  (Reference  37)  and  the  International  Test 
and  Evaluation  Association  (1'1'EA)  have  addressed 
the  issues  involved  in  the  EA  and  testing  of  Com¬ 
mand,  Control,  and  Communications  (C3)  systems. 
The  ITEA  study  illustrated  EA  in  Figure  20-4  and 
stated  that: 

With  regard  to  the  tester’s  role  in  EA,  the 
study  group  concluded  that  iterative  test 
and  evaluation  is  essential  for  success  in 
an  evolutionary  acquisition.  The  tester  must 
become  involved  early  in  the  acquisition 
process  and  contribute  throughout  the  de¬ 
velopment  and  fielding  of  the  core  and  the 
subsequent  increments.. ..The  testers  con¬ 
tribute  to  the  requirements  process  through 
feedback  of  test  results  to  the 
user.. .and.. .must  judge  the  ability  of  the 
system  to  evolve  (Reference  1 14). 

The  testing  of  EA  systems  presents  the  tester  with 
a  unique  challenge  as  the  core  system  must  be 
tested  during  fielding  and  the  first  increment  be¬ 
fore  the  core  testing  is  completed.  This  could  lead 
to  a  situation  where  the  tester  has  three  or  four 


tests  ongoing  on  various  increments  of  the  same 
system.  The  PM  must  insist  that  the  testing  for 
EA  systems  be  carefully  planned  to  ensure  the 
test  data  is  shared  by  all  and  there  is  a  minimum 
of  repetition  or  duplication  in  testing. 

20.4.2  Radio  Vulnerability 

The  Radio  Vulnerability  Analysis  (RVAN)  meth¬ 
odology  is  for  assessing  the  anti-jam  capability  and 
limitations  of  Radio  Frequency  (RF)  data  links 
when  operating  in  a  hostile  Electronic  Counter¬ 
measures  (ECM)  environment.  The  RVAN 
evolved  from  the  test  methodologies  developed 
for  an  Office  of  the  Secretary  of  Defense  (OSD)- 
chartered  Joint  Test  on  Data  Link  Vulnerability 
Analysis  (DVAL).  In  1983,  OSD  directed  the 
Services  to  apply  the  DVAL  methodology  to  all 
new  data  links  being  developed. 

The  purpose  of  the  DVAL  methodology  is  to  iden¬ 
tify  and  quantify  the  anti-jam  capabilities  and  vul¬ 
nerabilities  of  a  RF  data  link  operating  in  a  hos¬ 
tile  ECM  environment.  The  methodology  is  ap¬ 
plied  throughout  the  acquisition  process  and  per¬ 
mits  early  identification  of  needed  design  modifi¬ 
cations  to  reduce  identified  ECM  vulnerabilities. 
The  following  four  components  determine  a  data 
link’s  Electronic  Warfare  (EW)  vulnerability: 

(1)  The  susceptibility  of  a  data  link;  i.e.,  the 
receiver’s  performance  when  subjected  to 
intentional  threat  ECM; 

(2)  The  interceptibility  of  the  data  link;  i.e.,  the 
degree  to  which  the  transmitter  could  be  in¬ 
tercepted  by  enemy  intercept  equipment; 

(3  The  accessibility  of  the  data  link;  i.e.,  the 
likelihood  that  a  threat  jammer  could  de¬ 
grade  the  data  link’s  performance; 

(4)  The  feasibility  that  the  enemy  would  inter¬ 
cept  and  jam  the  data  link  and  successfully 
degrade  its  performance. 
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Figure  20-4.  The  Evolutionary  Acquisition  Process 


The  analyst  applying  the  DVAL  methodology  will 
require  test  data;  and  the  Test  Manager  (TM)  of 
the  C3I  system,  of  which  the  data  link  is  a  com¬ 
ponent,  will  be  required  to  provide  this  data.  The 
DVAL  joint  test  methodologies  and  test  results  are 
on  file  as  part  of  the  Joint  Test  Library  being  main¬ 
tained  by  the  USAF  Operational  Test  and  Evalu¬ 
ation  Center  (AFOTEC),  Kirtland  AFB,  New 
Mexico. 

20.5  T&E  OF  SURVEILLANCE  AND 
RECONNAISSANCE  SYSTEMS 

Intelligence,  Surveillance,  and  Reconnaissance 
(ISR)  capabilities  provide  the  requisite  battlespace 
awareness  tools  for  U.S.  Forces  to  take  and  hold 
the  initiative,  increase  operating  tempo,  and  con¬ 


centrate  power  at  the  time  and  place  of  their  choos¬ 
ing.  These  vital  capabilities  are  achieved  through 
highly  classified  sensor  systems  ranging  from  sat¬ 
ellites,  aircraft,  maritime  vessels,  electronic  inter¬ 
cept,  and  the  soldier  in  the  field  to  the  systems 
required  to  analyze  that  data,  synthesize  it  into 
usable  information,  and  disseminate  that  informa¬ 
tion  in  a  timely  fashion  to  the  warfighter.  As  a 
general  rule,  ISR  systems  are  considered  to  be 
Joint  Systems. 

Because  of  the  multifaceted  nature  of  ISR  pro¬ 
grams,  the  classified  nature  of  how  data  is  gath¬ 
ered  in  the  operational  element,  test  planning  to 
verify  effectiveness  and  suitability  can  be  com¬ 
plex.  Adding  to  that  inherent  complexity  is  the 
variable  nature  of  organizational  guidance  direc- 
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tive  upon  the  tester.  While  the  broad  management 
principles  enunciated  by  Department  of  Defense 
Directive  (DoDD)  5000.1,  The  Defense  Acquisi¬ 
tion  System,  12  May  2003,  will  apply  to  highly 
sensitive  classified  systems  and  cryptographic  and 
intelligence  programs,  the  detailed  guidance  con¬ 
tained  in  Defense  Acquisition  Guidebook  applies 
to  Major  Defense  Acquisition  Programs  (MDAPs) 
and  Major  Automated  Information  Systems 
(MAISs).  Many  ISR  programs  fall  below  this 
threshold  and  the  wise  TM  should  anticipate  that 
several  agencies  will  have  taken  advantage  of  this 
opening  to  tailor  organizational  guidance. 

Key  issues  for  the  T&E  of  ISR  systems  to  con¬ 
sider  include  compliance  verification  with  the  CII 
requirements  contained  in  DoDD  4630.5,  In¬ 
teroperability  and  Supportability  of  Information 
Technology  (IT)  and  National  Security  Systems 
(NSS),  5  May  2004,  Chairman  of  the  Joint  Chiefs 
of  Staff  Instruction  (CJCSI)  3170.01D,  Joint 
Capabilities  Integration  and  Development  Sys¬ 
tem,  12  May  2004,  and  CJCS  Instruction  (CJCSI) 
62 12.0 1C,  Interoperability  and  Supportability  of 
Information  Technology  and  National  Security 
Systems,  20  November  2003,  as  certified  by  the 
JITC;  completion  of  the  system  security  certifi¬ 
cation  required  prior  to  processing  classified  or 
sensitive  material,  verification,  and  documenta¬ 
tion  of  required  support  from  interfaced  C4ISR 
systems  in  the  Information  Support  Plan  (ISP) 


to  ensure  the  availability  and  quality  of  required 
input  data,  characterization  of  the  maturity  of  mis¬ 
sion  critical  software,  finalization  of  the  range  of 
human  factors  analysis,  and  consideration  of  In¬ 
formation  Operations  vulnerabilities/capabilities.  In 
addition  to  this  partial  listing,  many  of  these  sys¬ 
tems  will  operate  inside  a  matrix  of  ISR  system 
architectures  which  must  be  carefully  considered 
for  test  planning  purposes.  As  a  final  issue, 
Advanced  Concept  Technology  Demonstration 
(ACTD)  programs  are  being  used  to  quickly  de¬ 
liver  capability  to  the  user  because  of  the  critical 
and  time  sensitive  nature  of  many  ISR  require¬ 
ments.  The  TM  must  carefully  consider  structur¬ 
ing  the  T&E  effort  in  light  of  the  inherent  nature 
of  ACTD  programs. 

20.6  SUMMARY 

The  EC  systems  must  be  tested  under  conditions 
representative  of  the  dense  threat  signal  environ¬ 
ments  in  which  they  will  operate.  The  C4ISR  sys¬ 
tems  must  be  tested  in  representative  environments 
where  their  interaction  and  responsiveness  can  be 
demonstrated.  The  solution  for  the  tester  is  an  in¬ 
tegrated  approach  using  a  combination  of  analyti¬ 
cal  models,  computer  simulations,  hybrid  labora¬ 
tory  simulators  and  test  beds,  and  actual  field  test¬ 
ing.  The  tester  must  understand  these  test  tech¬ 
niques  and  resources  and  apply  them  in  EC  and 
C4ISR  T&E. 
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MULTI-SERVICE  TESTS 


21.1  INTRODUCTION 

This  chapter  discusses  the  planning  and  manage¬ 
ment  of  a  multi-Service  test  program.  A  multi-Ser- 
vice  test  program  is  conducted  when  a  system  is 
to  be  acquired  for  use  by  more  than  one  Service 
or  when  a  system  must  interface  with  equipment 
of  another  Service  (Joint  program).  A  multi-Ser- 
vice  test  program  should  not  be  confused  with  the 
Office  of  the  Secretary  of  Defense  (OSD)-spon- 
sored,  non-acquisition  oriented  Joint  Test  and 
Evaluation  (JT&E)  program.  A  brief  description 
of  the  JT&E  program  is  provided  in  Chapter  6. 
The  Defense  Acquisition  Guidebook  provides 
guidance  (Chapter  7)  on  management  of  joint 
acquisition  programs. 

21.2  BACKGROUND 

The  Defense  Acquisition  Guidebook  states:  The 
Component  Acquisition  Executive  (CAE)  of  a 
designated  acquisition  agent  given  acquisition  re¬ 
sponsibilities  shall  utilize  the  acquisition  and  test 
organizations  and  facilities  of  the  Military  Depart¬ 
ments  to  the  maximum  extent  practicable;  A  des¬ 
ignated  joint  program  shall  have  . .  .one  integrated 
test  program  and  one  set  of  documentation  (TEMP 
[Test  and  Evaluation  Master  Plan])  and  reports; 
The  MDA  [Milestone  Decision  Authority]  shall 
designate  a  lead  OTA  [Operational  Test  Agency] 
to  coordinate  all  OT&E  [Operational  Test  and 
Evaluation].  The  lead  OTA  shall  produce  a  single 
operational  effectiveness  and  suitability  report  for 
the  program;  Unless  statute,  the  MDA,  or  a  MOA 
[Memorandum  of  Agreement]  signed  by  all  DoD 
[Department  of  Defense]  components  directs  oth¬ 
erwise,  the  lead  executive  component  shall  bud¬ 
get  for  and  manage  the  common  RDT&E 


[Research,  Development,  Test  and  Evaluation]  funds 
for  assigned  joint  programs;  and,  Individual  compo¬ 
nents  shall  budget  for  their  unique  requirements. 

Formulation  of  a  multi-Service  Test  and  Evalua¬ 
tion  (T&E)  program  designates  the  participants  in 
the  program  and  gives  a  Lead  Service  responsi¬ 
bility  for  preparing  a  single  report  concerning  a 
system’s  operational  effectiveness  and  suitability. 
(The  Lead  Service  is  the  Service  responsible  for 
the  overall  management  of  a  Joint  Acquisition  pro¬ 
gram.  A  “Supporting  Service”  is  a  Service  desig¬ 
nated  as  a  participant  in  the  system  acquisition.) 

A  multi-Service  T&E  program  may  include  ei¬ 
ther  Development  Test  and  Evaluation  (DT&E) 
or  Operational  Test  and  Evaluation  (OT&E)  or 
both.  The  Service’s  OTAs  have  executed  a  for¬ 
mal  on  multi-Service  OT&E  (Reference  34)  that 
provides  a  framework  for  the  conduct  of  a  multi- 
Service  Operational  Test  (OT)  program.  The  De¬ 
fense  Information  Systems  Agency  (DISA)  has 
signed  the  MOA  in  relation  to  the  Joint  Interoper¬ 
ability  Test  and  certification  for  multi-Service 
T&E.  The  MOA  includes  a  glossary  of  common 
and  Service  peculiar  terms  used  in  multi-Service 
T&E.  Air  Force  Instruction  99-103  and  the  De¬ 
partment  of  the  Army  Pamphlet  73-1  provide  guid¬ 
ance  for  procedures  followed  in  a  multi-Service 
T&E  program.  Generally  the  process  includes: 

(1)  In  a  multi-Service  acquisition  program,  T&E 
is  planned  and  conducted  according  to  Lead 
Service  regulations.  The  designated  Lead 
Service  will  have  the  overall  responsibility 
for  management  of  the  multi-Service  pro¬ 
gram  and  will  ensure  that  supporting  Ser¬ 
vice  requirements  are  included.  If  another 


Service  has  certain  unique  T&E  require¬ 
ments,  testing  for  these  unique  requirements 
may  be  planned,  funded,  and  conducted  ac¬ 
cording  to  that  Service’s  regulations. 

(2)  Participating  Services  will  prepare  reports  in 
accordance  with  their  respective  regulations. 
The  Lead  Service  will  prepare  and  coordi¬ 
nate  a  single  DT&E  report  and  a  single 
OT&E  report,  which  will  summarize  the  con¬ 
clusions  and  recommendations  of  each 
Service’s  reports.  Rationale  will  be  provided 
to  explain  any  significant  differences.  The 
individual  Service  reports  may  be  attached 
to  this  single  report. 

(3)  Deviations  from  the  Lead  Service  T&E 
regulations  may  be  accommodated  by  mu¬ 
tual  agreement  among  the  Services  involved. 


21.3  TEST  PROGRAM 
RESPONSIBILITIES 

The  Lead  Service  has  overall  management  respon¬ 
sibility  for  the  program.  It  must  ensure  that  sup¬ 
porting  Service  requirements  are  included  in  the 
formulation  of  the  basic  resource  and  planning 
documents. 

A  multi-Service  T&E  Integrated  Product  Team 
(IPT)  should  be  established  for  each  multi-Ser¬ 
vice  test  program.  Its  membership  consists  of  one 
senior  representative  from  each  participating  Ser¬ 
vice  or  agency  headquarters.  The  T&E  IPT  works 
closely  with  the  Program  Management  Office 
(PMO)  and  is  responsible  for  arbitrating  all  dis¬ 
agreements  among  Services  that  cannot  be  re¬ 
solved  at  the  working  level. 


(1)  USED  FOR  COMPLEX  PROGRAMS  WITH  MANY  PARTICIPANTS 

SOURCE:  Memorandum  of  Agreement  on  Multi-Service  OT&E  and  Joint  T&E,  Aug  2003. 

Figure  21-1 .  Simple  Multi-Service  OT&E  Test  Team  Composition 
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Resource  requirements  are  documented  in  the 
TEMP.  Each  participating  Service  is  directed  to 
budget  for  the  testing  necessary  to  accomplish 
its  assigned  test  objectives  and  for  the  participa¬ 
tion  of  its  personnel  and  equipment  in  the  entire 
test  program.  Separate  annexes  may  be  used  to 
address  each  Service’s  test  requirements.  Fund¬ 
ing  will  be  in  accordance  with  Public  Law  and 
DoD  7000. 14-R,  Vol.  02B,  Budget  Formulation 
and  Presentation,  Chapter  5,  “Research,  Devel¬ 
opment,  Test,  and  Evaluation  Appropriations,” 
June  2004,  DoD  Financial  Management 
Regulation. 

21.4  TEST  TEAM  STRUCTURE 

A  sample  test  team  structure  is  shown  in  Figure 
21-1.  As  shown  in  the  figure,  Service  test  teams 
work  through  a  Service  deputy  test  director  or 
senior  representative.  The  test  director  exercises 
test  management  authority  but  not  operational 
control  over  the  test  teams.  The  responsibilities 
include  integration  of  test  requirements  and  effi¬ 
cient  scheduling  of  test  events.  The  deputy  test 
directors  exercise  operational  control  or  test  man¬ 
agement  authority  over  their  Service  test  teams 
in  accordance  with  their  Service  directives.  Ad¬ 
ditionally,  they  act  as  advisers  to  the  test  direc¬ 
tor;  represent  their  Service’s  interests;  and  are 
responsible,  at  least  administratively,  for  resources 
and  personnel  provided  by  their  Services. 

21.5  TEST  PLANNING 

Test  planning  for  multi-Service  T&E  is  accom¬ 
plished  in  the  manner  prescribed  by  Lead  Service 
directions  and  in  accordance  with  the  following 
general  procedures  extracted  from  the  “Memoran¬ 
dum  of  Agreement  on  Multi- Service  OT&E 
[MOT&E]  and  Joint  T&E”: 

(1)  The  Lead  Service  T&E  agency  begins  the 
planning  process  by  issuing  a  call  to  the  sup¬ 
porting  Service  T&E  agencies  for  critical 
issues  and  test  objectives. 


(2)  The  Lead  Service  T&E  agency  consolidates 
the  objectives  into  a  list  and  coordinates  the 
list  with  the  supporting  Service  T&E 
agencies. 

(3)  The  Lead  Service  T&E  agency  accommo¬ 
dates  supporting  Service  T&E  requirements 
and  input  in  the  formal  coordination  action 
of  the  TEMP. 

(4)  Participating  T&E  agency  project  officers 
assign  responsibility  for  the  accomplishment 
of  test  objectives  (from  the  consolidated  list) 
to  each  T&E  agency.  These  assignments  are 
made  in  a  mutually  agreeable  manner.  Each 
agency  is  then  responsible  for  resource  iden¬ 
tification  and  accomplishment  of  its  assigned 
test  objectives  under  the  direction  of  the 
Lead  Service  T&E  agency. 

(5)  Each  participating  agency  prepares  the  por¬ 
tion  of  the  overall  test  plan(s)  for  its  assigned 
objectives,  in  the  Lead  Service  test  plan(s) 
format,  and  identifies  its  data  needs. 

(6)  The  Lead  Service  T&E  agency  prepares  the 
multi-Service  T&E  test  plan(s),  consolidat¬ 
ing  the  input  from  all  participating  agencies. 

21.6  DEFICIENCY  REPORTING 

In  a  multi-Service  T&E  program,  a  deficiency  re¬ 
port  is  a  report  of  any  condition  that  reflects  ad¬ 
versely  on  the  item  being  tested  and  that  must  be 
reported  outside  the  test  team  for  corrective  ac¬ 
tion.  The  deficiency  reporting  system  of  the  Lead 
Service  is  normally  used.  All  members  of  the 
multi-Service  test  team  will  report  deficiencies 
through  their  Service’s  system. 

Items  undergoing  test  will  not  necessarily  be  used 
by  each  of  the  Services  for  identical  purposes.  As 
a  result,  a  deficiency  considered  disqualifying  by 
one  Service  is  not  necessarily  disqualifying  for  all 
Services.  Deficiency  reports  of  a  disqualifying  na¬ 
ture  must  include  a  statement  by  the  concerned 
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Service  of  why  the  deficiency  has  been  so  classified. 
It  also  includes  statements  by  the  other  Services 
as  to  whether  or  not  the  deficiency  affects  them 
significantly. 

If  one  of  the  participating  Services  identifies  a 
deficiency  that  it  considers  as  warranting  termina¬ 
tion  of  the  test,  the  circumstances  are  reported 
immediately  to  the  test  director. 

21.7  TEST  REPORTING 

The  following  test-reporting  policy  applies  to  multi- 
Service  IOT&E: 

(1)  The  Lead  OTA  will  prepare  and  coordi¬ 
nate  the  report  reflecting  the  system’s  op¬ 
erational  effectiveness  and  suitability  for 
each  Service.  Interim  reports  will  not  nor¬ 
mally  be  prepared. 

(2)  It  will  synergize  the  different  operational 
requirements  and  operational  environments 
of  the  involved  Services. 

(3)  It  will  state  findings,  put  those  findings  into 
perspective,  and  present  rationale  why  there 
is  or  is  not  consensus  on  the  utility  of  the 
system. 

(4)  All  participating  OTAs  will  sign  the  report. 

(5)  Each  Participating  OTA  may  prepare  an 
independent  evaluation  report  or  final  test 


report,  as  required,  in  its  own  format  and 
process  that  report  through  normal  Service 
channels. 

(6)  The  Lead  OTA  will  ensure  that  all  separate 
participating  Service  independent  evaluation 
reports/test  reports  are  appended  to  the  over¬ 
all  final  report  prepared  by  the  Lead  OTA 
for  submission  to  the  MDA. 

(7)  A  single  integrated  multi-Service  report  will 
be  submitted  90  calendar  days  after  the  offi¬ 
cial  end  of  test  is  declared,  but  not  later  than 
45  days  prior  to  a  milestone  decision. 

21.8  SUMMARY 

Multi-Service  test  programs  are  conducted  by  two 
or  more  Services  when  a  system  is  to  be  acquired 
for  employment  by  more  than  one  Service  or 
when  a  system  must  interface  with  equipment  of 
another  Service.  Test  procedures  for  multi-Ser- 
vice  T&E  follow  those  of  the  designated  Lead 
Service,  with  mutual  agreements  resolving  areas 
where  deviations  are  necessary.  The  MOA  on 
OT&E  procedures  provides  guidance  to  the  OTA. 
Care  must  be  exercised  when  integrating  test  re¬ 
sults  and  reporting  discrepancies  since  items  un¬ 
dergoing  testing  may  be  used  for  different  pur¬ 
poses  in  different  Services.  Close  coordination 
is  required  to  ensure  that  an  accurate  summary 
of  the  developing  system’s  capabilities  is  pro¬ 
vided  to  Service  and  DoD  decision  authorities. 
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INTERNATIONAL  TEST  AND 
EVALUATION  PROGRAMS 


22.1  INTRODUCTION 

This  chapter  discusses  Test  and  Evaluation  (T&E) 
from  an  international  perspective.  It  describes  the 
Office  of  the  Secretary  of  Defense  (OSD)-spon- 
sored  Foreign  Comparative  Test  (FCT)  Program 
(10  U.S.C.  2350)  and  the  International  Test  Op¬ 
erations  Procedures  (ITOPs).  Factors  that  bear  on 
the  T&E  of  multinational  acquisition  programs  are 
discussed  also. 

22.2  FOREIGN  COMPARATIVE  TEST 
PROGRAM 

22.2.1  Program  Objective 

The  FCT  Program  is  designed  to  support  the 
evaluation  of  a  foreign  nation’s  weapons  system, 
equipment  or  technology  in  terms  of  its  potential 
to  meet  a  valid  requirement  of  one  or  more  of  the 
U.S.  Armed  Services.  Additional  goals  of  the  FCT 
program  include  avoiding  unnecessary  duplication 
in  development,  enhancing  standardization  and 
interoperability,  and  promoting  international  tech¬ 
nology  exchanges.  The  FCT  program  is  not  in¬ 
tended  for  use  in  exploiting  threat  systems  or  for 
intelligence  gathering.  The  primary  objective  of 
the  program  is  to  support  the  U.S.  national  policy 
of  encouraging  international  armaments  coopera¬ 
tion  and  to  reduce  the  costs  of  Research  and  De¬ 
velopment  (R&D).  Policy  and  procedures  for  the 
execution  of  the  program  were  originally  docu¬ 
mented  in  Department  of  Defense  (DoD)  5000.3- 
M-2,  Foreign  Comparative  Testing  Program  Pro¬ 
cedures  Manual,  January  1994,  but  now  can  be 
found  in  the  Foreign  Comparative  Test  Handbook 
fhttp  ://www.  acq  .osd.mil/cto)  and  at  DoD  Instruc¬ 
tion  (DoDI)  5000.2,  Operation  of  the  Defense 


Acquisition  System,  12  May  2003,  Enclosure  5, 
Integrated  T&E. 

22.2.2  Program  Administration 

Foreign  Weapons  Evaluation  (FWE)  activities  and 
responsibilities  are  assigned  to  the  Director,  Com¬ 
parative  Test  Office  (CTO),  (Director  of  Defense 
Research  and  Engineering  (DDR&E))  OSD.  Each 
year,  sponsoring  military  Services  forward  Can¬ 
didate  Nomination  Proposals  (CNPs)  for  systems 
to  be  evaluated  under  the  FCT  program  to  the 
Director,  CTO.  The  Services  are  encouraged  to 
prepare  and  submit  a  CNP  whenever  a  promising 
candidate  that  appears  to  satisfy  a  current  or  po¬ 
tential  Service  requirement  is  found.  A  CNP  must 
contain  the  information  as  required  by  the  FCT 
Handbook. 

The  fundamental  criterion  for  FCT  program  se¬ 
lection  is  the  candidate  system’s  potential  to  sat¬ 
isfy  operational  or  training  requirements  that  exist 
or  are  projected.  Its  possible  contribution  to  the 
U.S.  technology  base  is  considered  also.  Addi¬ 
tional  factors  influencing  candidate  selection  in¬ 
clude:  candidate  maturity,  available  test  data,  multi- 
Service  interest,  existence  of  a  statement  of  op¬ 
erational  requirement  need,  potential  for  subse¬ 
quent  procurement,  sponsorship  by  U.S. -based  lic¬ 
ensee,  realistic  evaluation  schedule  cost,  DoD 
component  OSD  evaluation  cost-sharing  proposal, 
and  preprogrammed  procurement  funds.  For  tech¬ 
nology  evaluation  programs  within  the  FCT  pro¬ 
gram,  the  candidate  nomination  proposal  must 
address  the  specific  arrangements  under  which  the 
United  States  and  foreign  participants  (govern¬ 
ments,  armed  forces,  corporations)  will  operate. 
These  may  include  govemment-to-govemment 
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Memoranda  of  Agreement  (MOA),  private 
industry  licensing  agreements,  data  exchange 
agreements  and/or  cooperative  technology  ex¬ 
change  programs. 

FWE  activities  are  funded  by  OSD  and  executed 
by  the  Service  with  the  potential  need  for  the  sys¬ 
tem.  Points  of  contact  at  the  headquarters  level  in 
each  Service  monitor  the  conduct  of  the  programs. 
Work  is  performed  in  laboratories  and  test  centers 
throughout  the  country.  Systems  evaluated  recently 
under  the  FCT  program  include  millimeter  wave 
communications  equipment,  chemical  defense 
equipment,  gunnery  devices,  maritime  decoys  and 
navigational  systems.  The  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logis¬ 
tics  (USD(AT&L))  shall  notify  Congress  a  mini¬ 
mum  of  thirty  days  prior  to  the  commitment  of 
funds  for  initiation  of  new  FCT  evaluations. 
(DoDI  5000.2) 

22.3  NATO  COMPARATIVE  TEST 
PROGRAM 

The  North  Atlantic  Treaty  Organization  (NATO) 
Comparative  Test  Program  has  been  integrated 
with  the  FCT  program.  It  was  created  by  an  act 
of  the  Congress  in  the  Fiscal  Year  (FY)86  De¬ 
fense  Authorization  Bill.  The  program  supported 
the  evaluation  of  NATO  nations’  weapons  sys¬ 
tems,  equipment  and  technology  and  assessed  their 
suitability  for  use  by  U.S.  forces.  The  selection 
criteria  for  the  NATO  Comparative  Test  Program 
were  essentially  the  same  as  for  the  FCT  program. 
The  exception  was  that  the  equipment  must  be 
produced  by  a  NATO  member  nation  and  be  con¬ 
sidered  as  an  alternative  to  a  system  that  was  ei¬ 
ther  in  a  late  stage  of  development  in  the  United 
States  or  that  offered  a  cost,  schedule  or  perfor¬ 
mance  advantage  over  U.S.  equipment.  In  addi¬ 
tion,  the  NATO  Comparative  Test  Program  re¬ 
quired  that  notification  be  sent  to  the  Armed  Ser¬ 
vices  and  Appropriations  Committees  of  the 
House  of  Representatives  and  Senate  before 
funds  were  obligated.  With  this  exception,  the 
NATO  Comparative  Test  Program  followed  the 


same  nomination  process  and  administrative  pro¬ 
cedures.  Guidelines  for  the  program  were  also 
contained  in  the  FCT  Handbook. 

Examples  of  proposals  funded  under  the  NATO 
Comparative  Test  Program  included  T&E  of  a 
German  mine  reconnaissance  and  detection  sys¬ 
tem  for  the  Army,  a  United  Kingdom-designed 
mine  hunter  for  the  Navy,  and  the  Norwegian 
Penguin  missile  system  for  the  Air  Force.  Accord¬ 
ing  to  the  FY88  Report  of  the  Secretary  of  De¬ 
fense  (SECDEF)  to  the  Congress,  the  program 
generated  considerable  interest  among  NATO  al¬ 
lied  nations  and  became  a  primary  way  of  pro¬ 
moting  armaments  cooperation  within  NATO. 

Problems  associated  with  testing  foreign  weapons 
normally  stem  from  politics,  national  pride,  and  a 
lack  of  previous  test  data.  When  foreign  compa¬ 
nies  introduce  weapon  systems  for  testing,  they 
often  will  attempt  to  align  the  U.S.  military/ 
congressional  organizations  with  their  systems.  For 
example,  when  a  foreign  nation  introduced  an  an¬ 
titank  weapon  to  the  Army,  they  did  so  by  having 
a  U.S.  Senator  write  the  Army  stating  a  need  for 
the  system.  Attached  to  the  letter  was  a  document 
containing  doctrine  to  employ  the  system  and  a 
test  concept  to  use  when  evaluating  the  system. 
Systems  tested  in  the  NATO  Comparative  Test 
Program  often  become  involved  in  national  pride. 
The  test  community  must  be  careful  not  to  allow 
national  pride  to  be  a  driving  force  in  the  evalua¬ 
tion.  At  times,  the  9mm  pistol  competition  in 
NATO  resembled  an  international  soccer  match, 
with  each  competing  nation  cheering  for  their  pis¬ 
tol  and  many  other  nations  selecting  sides.  Evalu¬ 
ating  the  9mm  pistol  was  difficult  because  of  these 
forces.  Thus,  U.S.  testers  must  make  every  effort 
to  obtain  all  available  test  data  on  foreign  systems. 
These  data  can  be  used  to  help  validate  the  evolv¬ 
ing  test  data  and  additional  test  data  during  the 
evaluation. 
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22.4  T&E  MANAGEMENT  IN 

MULTINATIONAL  PROGRAMS 

22.4.1  Compatibility  With  Allies 

Rationalization,  Standardization,  and  Interoperabil¬ 
ity  (RSI)  have  become  increasingly  important  el¬ 
ements  in  the  materiel  acquisition  process.  Public 
Law  94-361,  passed  on  July  14,  1976,  requires 
that  “equipment  for  use  of  personnel  of  the  Armed 
Forces  of  the  United  States  stationed  in  Europe 
under  the  terms  of  the  North  Atlantic  Treaty  should 
be  standardized  or  at  least  interoperable  with 
equipment  of  other  members  of  the  North  Atlan¬ 
tic  Treaty  Organization.”  Program  Managers  (PMs) 
and  Test  Managers  (TMs)  must,  therefore,  be  fully 
aware  of  any  potential  international  applications 
of  the  systems  for  which  they  are  responsible.  Two 
guidebooks  published  by  the  Defense  Acquisition 
University  are  valuable  compendiums  of  informa¬ 
tion  for  the  PM  of  a  developing  system  with  po¬ 
tential  multinational  applications.  ( Comparison  of 
the  Defense  Acquisition  Systems  of  France,  Great 
Britain,  Germany  and  the  United  States,  Septem¬ 
ber  1999;  and  Comparison  of  the  Defense  Acqui¬ 
sition  Systems  of  Australia,  Japan,  South  Korea, 
Singapore  and  the  United  States,  July  2000.)  Ad¬ 
ditionally,  on  file  at  the  DAU  David  D.  Acker 
Library,  is  the  research  report  “Operational  Test 
and  Evaluation  in  the  IDEA  Nations”  (1999)  com¬ 
paring  the  Operational  Test  and  Evaluation 
(OT&E)  processes  of  the  U.S.  with  those  of  Great 
Britain,  Germany  and  France.  This  report  con¬ 
cluded  that  there  were  significant  differences  in 
what  was  considered  OT&E  by  the  four  coun¬ 
tries.  The  Defense  Acquisition  Guidebook  states 
that  Foreign  Military  Sales  (FMS)  of  U.S.  major 
systems  (ACAT  [Acquisition  Category  I,  II)  prior 
to  IOT&E  must  be  approved  by  the  USD(AT&L). 

Representatives  of  the  United  States,  United  King¬ 
dom,  France,  and  Germany  have  signed  a  MOA 
concerning  the  mutual  acceptability  of  each 
country’s  T&E  data.  This  agreement  seeks  to  avoid 
redundant  testing  by  documenting  the  extent  of 
understanding  among  involved  governments 


concerning  mutual  acceptability  of  respective  T&E 
procedures  for  systems  that  are  developed  in  one 
country  and  are  candidates  for  procurement  by  one 
or  more  of  the  other  countries.  Focal  points  for 
development  and  operational  testing  in  each  of  the 
countries  are  identified,  and  procedures  govern¬ 
ing  generation  and  release  of  T&E  data  are 
described  in  the  Memorandum  of  Understanding 
(MOU).  The  European  concept  of  OT&E  is  sig¬ 
nificantly  different  from  that  used  by  the  U.S. 
DoD. 

Early  and  thorough  planning  is  an  important  ele¬ 
ment  of  any  successful  T&E  program  but  is  even 
more  critical  in  a  multinational  program.  Agree¬ 
ment  must  be  reached  concerning  T&E  proce¬ 
dures,  data  requirements  and  methodology.  Dif¬ 
ferences  in  tactics,  battlefield  representations  and 
military  organizations  may  make  it  difficult  for  one 
nation  to  accept  another’s  test  data.  Therefore, 
agreement  must  be  reached  in  advance  concern¬ 
ing  the  operational  test  scenario  and  battlefield 
representation  that  will  be  used. 

22.4.2  International  Test  Operations 
Procedures 

The  Defense  Acquisition  Guidebook  (Chapter  2, 
C2.9.2  “International  Cooperation”)  states  “re¬ 
sults  of  T&E  of  systems  using  approved  Interna¬ 
tional  Test  Operations  Procedures  (ITOPs)  may 
be  accepted  without  repeating  the  testing.”  The 
ITOPs  are  documents  containing  standardized 
state-of-the-art  test  procedures  prepared  by  the 
cooperative  efforts  of  France,  Germany,  the  U.K., 
and  the  U.S.  Their  use  assures  high  quality,  effi¬ 
cient,  accurate,  and  cost  effective  testing.  The 
Director,  Operational  Test  and  Evaluation 
(DOT&E)  is  the  OSD  sponsor  for  providing  ba¬ 
sic  guidance  and  direction  to  the  ITOPs  pro¬ 
cesses.  The  ITOPs  program  is  intended  to 
shorten  and  reduce  costs  of  the  materiel  devel¬ 
opment  and  acquisition  cycle,  minimize  dupli¬ 
cate  testing,  improve  interoperability  of  U.S.  and 
Allied  equipment,  promote  the  cooperative  de¬ 
velopment  and  exchange  of  advanced  test 
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technology,  and  expand  the  customer  base.  Each 
Service  has  designated  an  ITOPs  point  of  con¬ 
tact.  The  Army  uses  the  Test  and  Evaluation  Man¬ 
agement  Agency  (TEMA),  in  the  Navy  it  is  the 
Director,  Navy  T&E  Division  (N-912),  and  the 
Air  Force  has  the  Chief,  Policy  and  Program  Di¬ 
vision  (AF/TEP).  The  Army,  who  initiated  the 
program  in  1979,  is  the  lead  Service.  A  total  of 
75  ITOPs  have  been  completed  and  published  in 
six  technical  areas  under  the  Four-Nation  Test  and 
Evaluation  MOU.  Additional  ITOPs  are  under 
development  by  active  working  committees,  (http:/ 
/www.dtc.army.mil/publications/topsindex.aspx) 
Completed  documents  are  submitted  to  the  De¬ 
fense  Technical  Information  Center  (DTIC)  for 
official  distribution. 

22.5  U.S.  AND  NATO  ACQUISITION 
PROGRAMS 

Some  test  programs  involve  combined  develop¬ 
ment  and  test  of  new  weapon  systems  for  U.S. 
and  other  NATO  countries.  In  these  programs, 
some  differences  from  the  regular  “way  of  doing 
things”  occur.  For  example,  the  formulation  of  the 
Request  for  Proposal  (RFP)  must  be  coordinated 
with  the  North  Atlantic  Program  Management 
Agency  (NAPMA);  and  their  input  to  the  State¬ 
ment  of  Work,  data  requirements,  operational  test 
planning  and  test  schedule  formulation  must  be 
included.  Also,  the  U.S.  Army  operational  user, 
Forces  Command  (FORSCOM),  must  be  involved 
in  the  OT  program.  Usually,  a  Multinational 
Memorandum  of  Understanding  (MMOU)  is  cre¬ 
ated  concerning  test  program  and  production  fund¬ 
ing,  test  resources,  test  team  composition,  use  of 
national  assets  for  testing,  etc. 

Nations  are  encouraged  to  use  the  data  that  an¬ 
other  nation  has  gathered  on  similar  test  programs 


to  avoid  duplication  of  effort.  For  example,  dur¬ 
ing  the  U.S.  and  NATO  Airborne  Warning  and 
Control  System  (AWACS)  Electronic  Support 
Measures  (ESM)  Program,  both  U.S.  and  NATO 
E-3As  will  be  used  for  test  aircraft  in  combined 
Development  Test  and  Evaluation  (DT&E)  and 
subsequent  OT&E.  Testing  will  be  conducted  in 
the  U.S.  and  European  theaters.  The  Joint  Test 
Force  will  be  composed  of  program  management 
office,  contractor,  U.S.  operational  users,  Air  Force 
Operational  Test  and  Evaluation  Center  (AFO- 
TEC),  Force  Command  (NATO  users),  and  lo¬ 
gistics  personnel  for  this  program.  A  Multinational 
Memorandum  of  Agreement  for  this  program  was 
created.  The  U.S.  program  is  managed  by  the 
AWACS  System  Program  Office,  and  the  NATO 
program  is  managed  by  the  NAPMA. 

22.6  SUMMARY 

The  procurement  of  weapon  systems  from  foreign 
nations  for  use  by  U.S.  Armed  Forces  can  pro¬ 
vide  the  following  advantages:  reduced  research 
and  development  costs,  faster  initial  operational 
capability,  improved  interoperability  with  friendly 
nations,  and  lower  procurement  costs  because  of 
economies  of  scale.  This  is  normally  a  preferred 
solution  to  user  requirements  before  attempting  to 
start  a  new  development.  Testing  such  systems 
presents  specific  challenges  to  accommodate  the 
needs  of  all  users.  OT&E  conducted  by  allied  and 
friendly  nations  is  not  as  rigorous  as  that  required 
by  U.S.  public  law  and  DoD  acquisition  regula¬ 
tions.  Such  testing  requires  careful  advance  plan¬ 
ning  and  systematic  execution.  Expectations  and 
understandings  must  be  well  documented  at  an 
early  stage  to  ensure  that  the  test  results  have  utility 
for  all  concerned. 
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COMMERCIAL  AND 
NONDEVELOPMENT  ITEMS 


23.1  INTRODUCTION 

Many  options  are  available  when  an  acquisition 
strategy  calls  for  a  material  solution  before  elect¬ 
ing  to  develop  a  new  system.  They  range  from 
the  preferred  solution  of  using  commercially 
available  products  from  domestic  or  international 
sources  to  the  least  preferred  option  of  a  tradi¬ 
tional  single  Service  new  Research  and  Devel¬ 
opment  (R&D)  program.  Between  these  two  ex¬ 
tremes  are  other  acquisition  strategies  that  call 
for  using  modified  systems  at  different  compo¬ 


nent  levels  and  unmodified  or  ruggedized  coop¬ 
erative  developments  to  various  extents.  (DoD 
Directive  (DoDD)  5000.1,  The  Defense  Acqui¬ 
sition  System,  12  May  2003.)  Figure  23-1  shows 
the  broad  spectrum  of  approaches  that  can  be 
taken  in  a  system  acquisition  and  provides  ex¬ 
amples  of  systems  that  have  been  developed  us¬ 
ing  each  approach.  The  PM  is  required  to  con¬ 
sider  this  spectrum  of  options  for  all  levels  of 
the  system  design.  (DoD  Instruction  (DoDI) 
5000.2,  Operation  of  the  Defense  Acquisition 
System,  12  May  2003.) 
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Source:  Army  Materiel  Command  Pamphlet  70-2,  AMC-TRADOC  Materiel  Acquisition  Handbook,  26  March  1987. 

Figure  23-1.  The  Spectrum  of  Technology  Maturity 


23.1.1  Definitions 

A  commercial  item  is  generally  defined  as  any 
item,  other  than  real  property,  that  is  of  a  type 
customarily  used  for  non-governmental  purposes 
and  that:  (1)  has  been  sold,  leased,  or  licensed  to 
the  general  public;  or,  (2)  has  been  offered  for 
sale,  lease,  or  license  to  the  general  public;  or  any 
item  that  evolved  through  advances  in  technol¬ 
ogy  or  performance  and  that  is  not  yet  available 
in  the  commercial  marketplace,  but  will  be 
available  in  the  commercial  marketplace  in  time 
to  satisfy  the  delivery  requirements  under  a  gov¬ 
ernment  solicitation.  Also  included  in  the  defini¬ 
tion  are  services  in  support  of  a  commercial  item, 
or  a  type  offered  and  sold  competitively  in  sub¬ 
stantial  quantities  in  the  commercial  marketplace 
based  on  established  catalog  or  market  prices  for 
specific  tasks  performed  under  standard  commer¬ 
cial  terms  and  conditions;  this  does  not  include 
services  that  are  sold  based  on  hourly  rates  with¬ 
out  an  established  catalog  or  market  price  for  a 
specific  service  performed. 

An  NDI  is  considered:  (1)  any  previously  devel¬ 
oped  item  of  supply  used  exclusively  for  govern¬ 
mental  purposes  by  a  federal  agency,  a  state,  or 
local  government,  or  a  foreign  government  with 
which  the  U.S.  has  a  mutual  defense  cooperation 
agreement;  (2)  any  item  described  in  (1)  that  re¬ 
quires  only  minor  modification  or  modifications 
of  the  type  customarily  available  in  the  commer¬ 
cial  marketplace  in  order  to  meet  the  requirements 
of  the  procuring  department  or  agency;  or  (3)  any 
item  described  in  (1)  or  (2)  solely  because  the  item 
is  not  yet  in  use. 

All  such  systems  are  required  to  undergo  techni¬ 
cal  and  Operational  Test  and  Evaluation  (OT&E) 
before  the  procurement  decision,  unless  the  deci¬ 
sion  authority  makes  a  definitive  decision  that  pre¬ 
vious  testing  or  other  data  (such  as  user/market 
investigations)  provide  sufficient  evidence  of  ac¬ 
ceptability  (Reference  16).  See  Federal  Acquisi¬ 
tion  Regulation  (FAR),  Section  2. 101  for  more  pre¬ 
cise  definitions  of  commercial  and  NDIs. 


23.1.2  Advantages  and  Disadvantages  of 
Commercial  and  NDI  Approaches 

The  use  of  commercial  and  NDI  offer  the  follow¬ 
ing  advantages: 

•  The  time  to  field  a  system  can  be  greatly  re¬ 
duced,  providing  a  quicker  response  to  the 
user’s  needs; 

•  Research  and  development  costs  or  total  own¬ 
ership  costs  may  be  reduced; 

•  State-of-the-art  technology  may  be  available 
sooner. 

Commercial  and  NDI  offers  the  following 
disadvantages: 

•  Acquisitions  are  difficult  to  standardize/inte¬ 
grate  with  the  current  fleet  equipment; 

•  Acquisitions  create  logistics  support  difficulties; 

•  Acquisitions  tend  not  to  have  comparable  com¬ 
petition;  therefore,  there  are  fewer  second 
sources  available; 

•  With  Commercial  and  NDI  acquisitions,  engi¬ 
neering  and  test  data  often  is  not  available. 

23.1.3  Application  of  Commercial  and  NDI 

Commercial  items  or  NDI  may  be  used  in  the  same 
environment  for  which  the  items  were  designed. 
Such  items  normally  do  not  require  development 
testing  prior  to  the  production  qualification  test  ex¬ 
cept  in  those  cases  where  a  contract  may  be 
awarded  to  a  contractor  who  has  not  previously 
produced  an  acceptable  finished  product  and  the 
item  is  assessed  as  high  risk.  In  that  case, 
preproduction  qualification  testing  would  be  re¬ 
quired  (Reference  16).  An  operational  assessment 
or  some  more  rigorous  level  of  OT&E  might  be 
appropriate. 
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Commercial  items  or  NDI  may  be  used  in  an  en¬ 
vironment  other  than  that  for  which  the  items  were 
designed.  Such  items  may  require  modifications 
in  hardware  and/or  software.  These  items  require 
testing  in  an  operational  environment,  pre- 
production  qualification  testing  (if  previous  testing 
resulted  in  item  redesign),  and  production  qualifi¬ 
cation  testing. 

Integration  of  commercial  items  or  NDI  into  a  new 
development  system  may  require  some  regression 
testing.  These  efforts  require  more  extensive  re¬ 
search,  development,  and  testing  to  achieve  ef¬ 
fective  operation  of  the  desired  system  configura¬ 
tion.  Testing  required  includes:  feasibility  testing 
in  a  military  environment,  preproduction  qualifi¬ 
cation  testing,  hardware/software  integration  test¬ 
ing,  operational  testing,  and  production  qualifica¬ 
tion  testing.  Given  the  variety  of  approaches  that 
may  be  employed,  it  is  imperative  that  the  acqui¬ 
sition  strategy  clearly  specifies,  with  the  agreement 
of  the  testing  authority,  the  level  of  testing  that 
will  be  performed  on  commercial  items  and  NDI 
systems  and  the  environment  in  which  those 
systems  will  be  tested. 

23.2  MARKET  INVESTIGATION  AND 
PROCUREMENT 

A  market  investigation  is  the  central  activity  lead¬ 
ing  to  the  program  initiation  decision  regarding 
the  use  of  a  commercial  item  or  NDI  acquisition 
strategy.  The  purpose  of  the  market  investigation 
is  to  determine  the  nature  of  available  products 
and  the  number  of  potential  vendors.  Market  in¬ 
vestigations  may  vary  from  informal  telephone 
inquiries  to  comprehensive  industry-wide  reviews. 
During  the  market  investigation,  sufficient  data 
must  be  gathered  to  support  a  definitive  decision, 
to  finalize  the  requirements  and  to  develop  an  ac¬ 
quisition  strategy  that  is  responsive  to  these  re¬ 
quirements.  Included  in  the  search  would  be  dual 
use  technologies  that  meet  military  needs,  yet  have 
sufficient  commercial  application  to  support  a 
viable  production  base.  An  assessment  of  Tech¬ 
nology  Readiness  Level  {Defense  Acquisition 


Guidebook,  Appendix  6)  will  provide  the  program 
office  with  insights  into  the  readiness  of  the  com¬ 
mercial  item  technologies  for  military  application. 

During  the  market  investigation,  a  formal  “request 
for  information”  process  may  be  followed  wherein 
a  brief  narrative  description  of  the  requirement  is 
published  and  interested  vendors  are  invited  to 
respond.  Test  samples  or  test  items  may  be  leased 
or  purchased  at  this  time  to  support  the  conduct 
of  operational  suitability  tests,  to  evaluate  the 
ability  of  the  equipment  to  satisfy  the  requirements 
and  to  help  build  the  functional  purchase  descrip¬ 
tion  or  system  specification.  This  type  of  prelimi¬ 
nary  testing  should  not  be  used  to  select  or  elimi¬ 
nate  any  particular  vendor  or  product  unless  it  is 
preceded  by  competitive  contracting  procedures 
(Reference  60). 

It  is  imperative  that  technical  and  operational 
evaluators  become  involved  during  this  early  stage 
of  any  commercial  item  or  NDI  procurement  and 
that  they  perform  an  early  assessment  of  the  ini¬ 
tial  issues.  The  evaluator  must  also  relate  these 
issues  to  test  and  evaluation  criteria  and  provide 
their  independent  evaluation  plans  and  reports  to 
the  decision  authorities  before  the  Milestone  B 
decision  review. 

23.3  COMMERCIAL  ITEM  AND  NDI 
TESTING 

23.3.1  General  Considerations 

Test  planning  for  commercial  and  nondevelopmen- 
tal  items  shall  recognize  commercial  testing  and 
experience,  but  nonetheless  determine  the  appro¬ 
priate  DT&E,  OT&E,  and  LFT&E  [Live  Fire  Test 
and  Evaluation]  needed  to  ensure  effective  per¬ 
formance  in  the  intended  operational  environment. 
(DoDI  5000.2)  The  Defense  Acquisition  Guide¬ 
book  suggests  that  “the  PM  shall  develop  an  ap¬ 
propriate  T&E  strategy  for  commercial  items  to 
include  evaluating  commercial  items  in  a  system 
test  bed,  when  practical;  focusing  test  beds  on 
high-risk  items;  and  testing  commercial  item 
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upgrades  for  unanticipated  side  effects  in  areas 
such  as  security,  safety,  reliability,  and  perfor¬ 
mance.”  T&E  must  be  considered  throughout  the 
acquisition  of  a  system  that  involves  commercial 
items  and  NDI. 

The  Program  Manager  (PM)  and  his/her  staff 
must  ensure  that  the  testing  community  is  fully 
involved  in  the  acquisition  from  the  start.  The 
amount  and  level  of  testing  required  depends  on 
the  nature  of  the  commercial  item  or  NDI  and 
its  anticipated  use;  it  should  be  planned  to  sup¬ 
port  the  design  and  decision  process.  At  a  mini¬ 
mum,  T&E  will  be  conducted  to  verify  integra¬ 
tion  and  interoperability  with  other  system  ele¬ 
ments.  All  commercial  item  and  NDI  modifica¬ 
tions  necessary  to  adapt  them  to  the  weapon  sys¬ 
tem  environment  will  also  be  subject  to  T&E. 
Available  test  results  from  all  commercial  and 
government  sources  will  determine  the  actual 
extent  of  testing  necessary.  For  example,  a  com¬ 
mercial  item  or  NDI  usually  encompasses  a  ma¬ 
ture  design.  The  availability  of  this  mature  de¬ 
sign  contributes  to  the  rapid  development  of  the 
logistics  support  system  that  will  be  needed.  In 
addition,  there  are  more  “production”  items  avail¬ 
able  for  use  in  a  test  program.  The  PM  and  his/ 
her  staff  must  remember  that  these  systems  also 
require  activity  in  areas  associated  with  tradi¬ 
tional  development  and  acquisition  programs.  For 
example,  training  and  maintenance  programs  and 
manuals  must  be  developed;  and  sufficient  time 
should  be  allowed  for  their  preparation. 

When  the  solicitation  package  for  a  commercial 
item  or  NDI  acquisition  is  assembled,  the  PM  must 
ensure  that  it  includes  the  following  T&E-related 
items: 

(1)  Approved  T&E  issues  and  criteria; 

(2)  A  requirement  that  the  contractor  provide  a 
description  of  the  testing  previously  per¬ 
formed  on  the  system,  including  test  proce¬ 
dures  followed,  data  and  results  achieved; 


(3)  Production  qualification  test  and  quality 
conformance  requirements; 

(4)  Acceptance  test  plans  for  the  system  and  its 
components. 

23.3.2  Testing  Before  Program  Initiation 

Since  an  important  advantage  of  using  a  commer¬ 
cial  item  or  NDI  acquisition  strategy  is  reduced 
acquisition  time,  it  is  important  that  testing  not  be 
redundant  and  that  it  is  limited  to  the  minimum 
effort  necessary  to  obtain  the  required  data.  Test¬ 
ing  can  be  minimized  by: 

(1)  Obtaining  and  assessing  contractor  test 
results; 

(2)  Obtaining  usage/failure  data  from  other 
customers; 

(3)  Observing  contractor  testing; 

(4)  Obtaining  test  results  from  independent  test 
organizations  (e.g.,  Underwriter’s  Laboratory); 

(5)  Verifying  selected  contractor  test  data. 

If  it  is  determined  that  more  information  is  needed 
after  the  initial  data  collection  from  the  above 
sources,  commercial  items  or  NDI  candidates  may 
be  bought  or  leased,  and  technical  and  operational 
tests  may  be  conducted. 

23.3.3  Testing  After  Program  Initiation 

All  testing  to  be  conducted  after  the  program  ini¬ 
tiation  milestone  decision  to  proceed  with  the 
commercial  item  or  NDI  acquisition  should  be 
described  in  the  Acquisition  Strategy  and  the  Test 
and  Evaluation  Master  Plan  (TEMP).  Develop¬ 
ment  testing  is  conducted  only  if  specific  infor¬ 
mation  that  cannot  be  satisfied  by  contractor  or 
other  test  data  sources  is  needed.  Operational  test¬ 
ing  is  conducted  as  needed.  The  Independent 
Operational  Test  and  Evaluation  (IOT&E)  agency 
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should  concur  in  any  decisions  to  limit  or  elimi¬ 
nate  operational  testing. 

Test  and  evaluation  continue  even  after  the  sys¬ 
tem  has  been  fielded.  This  testing  takes  the  form 
of  a  follow-on  evaluation  to  validate  and  refine: 
operating  and  support  cost  data;  Reliability,  Avail¬ 
ability,  and  Maintainability  (RAM)  characteristics; 
logistic  support  plans;  and  training  requirements, 
doctrine  and  tactics. 

23.4  RESOURCES  AND  FUNDING 

Programming  and  budgeting  for  a  commercial 
item  or  NDI  acquisition  present  a  special  chal¬ 
lenge.  Because  of  the  short  duration  of  the  acqui¬ 
sition  process,  the  standard  lead  times  required  in 
the  normal  Planning,  Programming,  Budgeting  and 
Execution  (PPBE)  system  cycle  may  be  unduly 
restrictive.  This  situation  can  be  minimized 
through  careful,  advanced  planning  and,  in  the 
case  of  urgent  requirements,  reprogramming/ 
supplemental  funding  techniques. 

Research,  Development,  Test,  and  Evaluation 
(RDT&E)  funds  are  normally  used  to  support  the 
conduct  of  the  Market  Investigation  Phase  and  the 
purchase  or  lease  of  candidate  systems/components 
required  for  T&E  purposes.  The  RDT&E  funds 
are  also  used  to  support  T&E  activities  such  as: 


modification  of  the  test  article;  purchase  of  speci¬ 
fications,  manufacturer’s  publications,  repair  parts, 
special  tools  and  equipment;  transportation  of  the 
test  article  to  and  from  the  test  site;  and  training, 
salaries  and  temporary  duty  costs  of  T&E  person¬ 
nel.  Procurement,  operations  and  maintenance 
funds  are  usually  used  to  support  production  and 
deployment  costs. 

One  chief  reason  for  using  a  commercial  item  or 
NDI  acquisition  strategy  is  reduced  overall  Total 
Ownership  Cost  (TOC).  Additional  cost  savings 
can  be  achieved  after  a  contract  has  been  awarded 
if  the  PM  ensures  that  incentives  are  provided  to 
contractors  to  submit  Value  Engineering  Change 
Proposals  (VECPs)  to  the  government  when  un¬ 
necessary  costs  are  identified. 

23.5  SUMMARY 

The  use  of  commercial  items  and  NDIs  in  a  sys¬ 
tem  acquisition  can  provide  considerable  time  and 
cost  savings.  The  testing  approach  used  must  be 
carefully  tailored  to  the  type  of  system,  levels  of 
modifications,  technology  risk  areas,  and  the 
amount  of  test  data  already  available.  The  T&E 
community  must  get  involved  early  in  the  process 
so  that  all  test  issues  are  adequately  addressed  and 
timely  comprehensive  evaluations  are  provided  to 
decision  authorities. 
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TESTING  THE 
SPECIAL  CASES 


24.1  INTRODUCTION 

This  chapter  covers  the  special  factors  and  alter¬ 
native  test  strategies  the  tester  must  consider  in 
testing  dangerous  or  lethal  weapons,  systems  that 
involve  one-of-a-kind  or  limited  production,  Ad¬ 
vanced  Concept  Technology  Demonstrations 
(ACTDs),  and  systems  with  high-cost  and/or  spe¬ 
cial  security  considerations.  Examples  include: 
chemical  and  laser  weapons;  ships;  space 
weapons;  and  missile  systems. 

24.2  TESTING  WITH  LIMITATIONS 

Certain  types  of  systems  cannot  be  tested  using 
relatively  standard  Test  and  Evaluation  (T&E)  ap¬ 
proaches  for  reasons  such  as  a  nonstandard  ac¬ 
quisition  strategy,  resource  limitations,  cost,  safety, 
or  security  constraints.  The  Test  and  Evaluation 
Master  Plan  (TEMP)  must  contain  a  statement  that 
identifies  “those  factors  that  will  preclude  a  full 
and  completely  realistic  operational  test...(IOT&E 
[Initial  Operational  Test  and  Evaluation]  and 
FOT&E  [Follow-on  Operational  Test  and  Evalu¬ 
ation]),”  such  as  inability  to  realistically  portray 
the  entire  threat,  limited  resources  or  locations, 
safety,  and  system  maturity.  The  impact  of  these 
limitations  on  the  test’s  Critical  Operational  Issues 
(COIs)  must  also  be  addressed  in  the  TEMP. 

Nonstandard  acquisition  strategies  are  often  used 
for  one-of-a-kind  or  limited  production  systems. 
Examples  of  these  include  space  systems;  missiles; 
and  ships.  For  one-of-a-kind  systems,  the  produc¬ 
tion  decision  is  often  made  prior  to  system  de¬ 
sign;  hence,  testing  does  not  support  the  traditional 
decision  process.  In  limited  production  systems, 
there  are  often  no  prototypes  available  for  test; 


consequently,  the  tester  must  develop  innovative 
test  strategies. 

The  tester  of  dangerous  or  lethal  systems,  like 
chemical  and  laser  weapons,  must  consider  vari¬ 
ous  safety,  health,  and  medical  factors  in  devel¬ 
oping  test  plans,  such  as: 

(1)  Provision  of  medical  facilities  for  pre-  and 
post-test  checkups  and  emergency  treatment; 

(2)  Need  for  protective  gear  for  participating/ 
observer  personnel; 

(3)  Approval  of  the  test  plan  by  the  Surgeon 
General; 

(4)  Restrictions  in  selection  of  test  participants 
(e.g.,  medical  criteria  or  use  of  only  volun¬ 
teer  troops); 

(5)  Restricted  test  locations; 

(6)  Environmental  Impact  Statements  (EISs). 

Also,  the  tester  must  allow  for  additional  plan¬ 
ning  time,  test  funds,  and  test  resources  to  accom¬ 
modate  such  factors. 

24.2.1  Chemical  Weapons  Testing 

The  testing  of  chemical  weapons  poses  unique 
problems,  because  the  tester  cannot  perform  ac¬ 
tual  open-air  field  testing  with  real  nerve  agents 
or  other  toxic  chemicals.  Since  the  United  States 
signed  and  ratified  the  Geneva  Protocol  of  1925, 
U.S.  policy  has  been  that  the  United  States  will 
never  be  the  first  to  use  lethal  chemical  weapons; 
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it  may,  however,  retaliate  with  chemical  weapons 
if  so  attacked.  In  addition  to  the  health  and  safety 
factors  discussed  in  the  last  paragraph,  test  issues 
the  chemical  weapons  tester  must  address  include: 

(1)  All  possible  chemical  reactions  due  to  varia¬ 
tions  such  as  moisture,  temperature,  pressure, 
and  contamination; 

(2)  Physical  behavior  of  the  chemical;  i.e.,  drop¬ 
let  size,  dispersion  density,  and  ground  con¬ 
tamination  pattern  when  used  operationally; 

(3)  Toxicity  of  the  chemical;  i.e.,  lethality  and 
duration  of  contamination  when  used 
operationally; 

(4)  Safety  of  the  chemical  weapon  during 
storage,  handling,  and  delivery; 

(5)  Decontamination  process. 

Addressing  all  of  these  issues  requires  a  combi¬ 
nation  of  laboratory  toxic  chamber  tests  and  open- 
air  field  testing.  The  latter  must  be  performed  using 
“simulants,”  which  are  substances  that  replicate 
the  physical  and  chemical  properties  of  the  agent 
but  with  no  toxicity. 

The  development  and  use  of  simulants  for  testing 
will  require  increased  attention  as  more  chemical 
weapons  are  developed.  Chemical  agents  can 
demonstrate  a  wide  variety  of  effects  depending 
on  such  factors  as  moisture,  temperature,  and  con¬ 
tamination.  Consequently,  the  simulants  must  be 
able  to  replicate  all  possible  agent  reactions;  it  is 
likely  that  several  simulants  would  have  to  be  used 
in  a  test  to  produce  all  predicted  agent  behaviors. 
In  developing  and  selecting  simulants,  the  tester 
must  thoroughly  understand  all  chemical  and 
physical  properties  and  possible  reactions  of  the 
agent. 

Studies  of  the  anticipated  reactions  can  be  per¬ 
formed  in  toxic-chamber  tests  using  the  real  agent. 
Here,  factors  such  as  changes  in  moisture. 


temperature,  pressure,  and  levels  of  impurity  can 
be  controlled  to  assess  the  agent’s  behavior.  But, 
the  tester  must  think  through  all  possible  environ¬ 
mental  conditions  in  which  the  weapon  could 
operate  so  all  cases  can  be  tested  in  the  laboratory 
chamber  with  the  real  agent.  For  example,  during 
development  testing  of  the  BIGEYE  chemical 
weapon,  it  was  found  that  higher-than-expected 
temperatures  due  to  aerodynamic  heating  caused 
pressure  buildup  in  the  bomb  body  that  resulted 
in  the  bomb  exploding.  This  caused  the  opera¬ 
tional  concept  for  the  BIGEYE  to  be  changed 
from  on-board  mixing  of  the  two  chemicals  to 
mixing  after  release  of  the  bomb. 

Tests  to  confirm  toxicity  must  be  conducted  using 
simulants  in  the  actual  environment.  Since  the 
agent’s  toxicity  is  dependent  on  factors  such  as 
droplet  size,  dispersion  density,  ground  contami¬ 
nation  pattern,  and  degradation  rate,  a  simulant 
that  behaves  as  the  agent  does  must  be  used  in 
actual  field  testing.  Agent  toxicity  is  determined 
in  the  lab. 

The  Services  publish  a  variety  of  technical  docu¬ 
ments  on  specific  chemical  test  procedures.  Docu¬ 
ments  such  as  the  U.S.  Army  Test  and  Evaluation 
Command  (TECOM)  Pamphlet  310-4,  Index  of 
Test  Operations  Logistics  Support:  Developmen¬ 
tal  Supportability  Test  and  Evaluation  Guide,  a 
bibliography  that  includes  numerous  reports  on 
chemical  testing  issues  and  procedures,  can  be  con¬ 
sulted  for  specific  documentation  on  chemical  test¬ 
ing. 

24.2.3  Laser  Weapons  Testing 

Many  new  weapon  systems  are  being  designed 
with  embedded  laser  range  finders  and  laser  des¬ 
ignators.  Because  of  the  danger  to  the  human  eye 
posed  by  lasers,  the  tester  must  adhere  to  special 
safety  requirements  and  utilize  special  configured 
geographic  locations  during  T&E.  For  instance, 
the  program  seeking  to  free-play  non-eye  safe  air¬ 
borne  or  ground  lasers  during  tests  involves  sig¬ 
nificant  precautions,  such  as  the  airspace  must  be 


24-2 


restricted  from  over-flight  during  active  testing  and 
guards  must  be  posted  to  prevent  anyone  (hikers, 
bikers,  off-road  vehicles,  equestrians)  accidentally 
venturing  into  the  area.  A  potential  solution  to  the 
safety  issue  is  to  develop  and  use  an  “eye-safe” 
laser  for  testing.  The  tester  must  ensure  that  eye- 
safe  lasers  produce  the  same  laser  energy  as  the 
real  laser  system. 

Another  concern  of  the  laser/directed  energy 
weapons  tester  is  the  accurate  determination  of 
energy  levels  and  location  on  the  target.  Measure¬ 
ments  of  the  energy  on  the  target  are  usually  con¬ 
ducted  in  the  laboratory  as  part  of  Development 
Test  (DT).  In  the  field,  video  cameras  are  often 
used  to  verify  that  a  laser  designator  did  indeed 
illuminate  the  target.  Such  determinations  are  im¬ 
portant  when  the  tester  is  trying  to  attribute 
weapon  performance  to  behavior  of  the  directed 
energy  weapon,  behavior  of  the  guidance  system, 
or  some  other  factor. 

A  bibliography  of  Army  test  procedures,  TECOM 
Pamphlet  310-4,  lists  several  documents  that  cover 
the  special  issues  associated  with  laser  testing. 

24.3  SPACE-SYSTEM  TESTING 

From  a  historical  perspective,  space-system  acqui¬ 
sition  has  posed  several  unique  problems  to  the 
test  process  (especially  the  operational  test  pro¬ 
cess)  that  generally  fall  into  four  categories:  lim¬ 
ited  quantities/high  cost,  “incremental  upgrade” 
approach  to  acquisition,  operating  environment 
(peacetime  and  wartime),  and  test  environment. 
Expanded  Air  Force  guidance  can  be  found  in 
AFM  99-113,  Space  Systems  Test  and  Evalua¬ 
tion  Process  (1  May  1996).  More  generic  guid¬ 
ance  is  in  National  Security  Space  (NSS)  Acqui¬ 
sition  Policy  03-01  (3  October  2003)  that  outlines 
a  streamlined  acquisition  framework. 

(1)  Limited  quantities/high  cost  —  Space  sys¬ 
tems  have  traditionally  involved  the  acquisi¬ 
tion  of  relatively  few  (historically,  less  than 
20)  systems  at  extremely  “high  per-unit 


costs”  (in  comparison  with  more  traditional 
military  systems).  The  high  per-unit  costs 
are  driven  by  a  combination  of  high 
transportation  costs  (launch  to  orbit),  high 
life-cycle  reliability  requirements  and  asso¬ 
ciated  costs  because  of  the  lack  of  an  “on- 
orbit”  maintenance  capability  and  the  high 
costs  associated  with  “leading  edge”  tech¬ 
nologies  that  tend  to  be  a  part  of  spacecraft 
design.  From  a  test  perspective,  this  serves 
to  drive  space-system  acquisition  strategy 
into  the  “nonstandard”  approach  addressed 
below.  The  problem  is  compounded  by  the 
“incremental  upgrade”  approach  to 
acquisition. 

(2)  Incremental  approach  to  acquisition  — 

Due  to  the  “limited  buy”  and  “high  per-unit 
cost”  nature  of  spacecraft  acquisition,  these 
systems  tend  to  be  procured  using  an  indi¬ 
vidual  increment  acquisition  strategy.  Under 
this  concept,  “the  decision  to  deploy”  is  of¬ 
ten  made  at  the  front  end  of  the  acquisition 
cycle;  and  the  first  prototype  to  be  placed  in 
orbit  becomes  the  first  operational  asset.  As 
early  and  follow-on  systems  undergo  ground 
and  on-orbit  testing  [either  Development  Test 
and  Evaluation  (DT&E)  or  Operational  Test 
and  Evaluation  (OT&E)],  discrepancies  are 
corrected  by  “increment  changes”  to  the  next 
system  in  the  pipeline.  This  approach  to  ac¬ 
quisition  can  perturb  the  test  process  as  the 
tester  may  have  no  formal  milestone  deci¬ 
sions  to  test  toward.  The  focus  must  change 
toward  being  able  to  influence  the  design  of 
(and  block  changes  to)  systems  further 
downstream  in  the  pipeline.  As  the  first  “on- 
orbit”  asset  usually  becomes  the  first  opera¬ 
tional  asset,  pressure  is  created  from  the  op¬ 
erational  community  to  expedite  (and  some¬ 
times  limit)  testing  so  a  limited  operational 
capability  can  be  declared  and  the  system  can 
begin  fulfilling  mission  requirements.  Once 
the  asset  “goes  operational,”  any  use  of  it  for 
testing  must  compete  with  operational  mis¬ 
sion  needs  —  a  situation  potentially  placing 
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the  tester  in  a  position  of  relatively  low  pri¬ 
ority.  Recognition  of  these  realities  and  care¬ 
ful  “early-on”  test  planning  can  overcome 
many  of  these  problems,  but  the  tester  needs 
to  be  involved  and  ready  much  earlier  in  the 
cycle  than  with  traditional  systems. 

(3)  Operating  environment  (peacetime  and 
wartime)  —  Most  currently  deployed  space 
systems  and  near-term  future  space  systems 
operate  in  the  military  support  arena  such  as 
tactical  waming/attack  assessment,  commu¬ 
nications,  navigation,  weather,  and  intelli¬ 
gence;  and  their  day-to-day  peacetime  oper¬ 
ating  environment  is  not  much  different  from 
the  wartime  operating  environment  except 
for  activity  level  (i.e.,  message  throughput, 
more  objects  to  track/see,  etc.).  Historically, 
space  has  been  a  relatively  benign  battlefield 
environment  because  of  technology  limita¬ 
tions  in  the  capability  of  potential  adversar¬ 
ies  to  reach  into  space  with  weapons.  But 
this  is  no  longer  valid.  This  combination  of 
support-type  missions  and  a  battlefield  envi¬ 
ronment  that  is  not  much  different  from  the 
peacetime  environment  has  played  a  definite 
role  in  allowing  systems  to  reach  limited 
operational  capability  without  as  much  dedi¬ 
cated  prototype  system-level  testing  as  seen 
on  other  systems.  This  situation  is  changing 
with  the  advent  of  concepts  like  the  Missile 
Defense  Agency  [MDA]  system  where  ac¬ 
tual  weapons  systems  (impact  anti-satellite 
and  laser)  may  be  in  operation,  and  day-to- 
day  peacetime  operations  will  not  mirror  the 
anticipated  battlefield  environment  as  closely. 
Likewise,  the  elevation  of  the  battlefield  into 
space  and  the  advancing  technologies  that 
allow  potential  adversaries  to  reach  into 
space  is  changing  the  thrust  of  how  space 
systems  need  to  be  tested  in  space.  The 
Department  of  Defense  (DoD)  should  an¬ 
ticipate  an  increased  need  for  dedicated  on- 
orbit  testing  on  a  type  of  space  range  where 
the  battlefield  environment  will  be  replicated 
can  be  anticipated  —  a  situation  similar  to 


the  dedicated  testing  done  today  on  test 
ranges  with  Army,  Navy,  and  Air  Force 
weapons. 

(4)  Test  environment  —  The  location  of  space 
assets  in  “remote”  orbits  also  compounds  the 
test  problem.  Space  systems  do  not  have  the 
ready  access  (as  with  ground  or  aircraft  sys¬ 
tems)  to  correct  deficiencies  identified  dur¬ 
ing  testing.  This  situation  has  driven  the 
main  thrust  of  testing  into  the  “prelaunch” 
ground  simulation  environment  where  dis¬ 
crepancies  can  be  corrected  before  the  sys¬ 
tem  becomes  inaccessible.  However,  as  men¬ 
tioned  previously,  when  space-system  mis¬ 
sions  change  from  a  war-support  focus  to  a 
war-fighting  focus,  and  the  number  of  sys¬ 
tems  required  to  do  the  mission  increases 
from  the  “high  reliability/limited  number” 
mode  to  a  more  traditional  “fairly  large  num¬ 
ber  buy”  mode,  future  space-system  testing 
could  be  expected  to  become  more  like  the 
testing  associated  with  current  ground,  sea, 
and  air  systems.  From  a  test  perspective,  this 
could  also  create  unique  “test  technology” 
requirements;  i.e.,  with  these  systems  we  will 
have  to  bring  the  test  range  to  the  operating 
system  as  opposed  to  bringing  the  system  to 
the  range.  Also,  because  the  space  environ¬ 
ment  tends  to  be  “visible  to  the  world”  (oth¬ 
ers  can  observe  our  tests  as  readily  as  we 
can),  unique  test  operations  security  meth¬ 
odologies  may  be  required  to  allow  us  to 
achieve  test  realism  without  giving  away 
system  vulnerabilities. 

In  summary,  current  and  near-term  future  space 
systems  have  unique  test  methodologies.  How¬ 
ever,  in  the  future,  space  operations  might  entail 
development/deployment  of  weapon  platforms  on 
orbit  with  lower  design-life  reliability  (because  of 
cost);  and  day-to-day  peacetime  operations  will 
not  mirror  the  wartime  environment.  Thus,  space- 
system  testing  requirements  may  begin  to  more 
closely  parallel  those  of  traditional  weapon 
systems. 
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24.4  OPERATIONS  SECURITY  AND  T&E 

Operations  Security  (OPSEC)  issues  must  be  con¬ 
sidered  in  all  test  planning.  Security  regulations 
and  contracting  documents  require  the  protection 
of  “sensitive  design  information  and  test  data” 
throughout  the  acquisition  cycle  by: 

(1)  Protecting  sensitive  technology; 

(2)  Eliminating  nonsecure  transmittal  data  on  and 
from  test  ranges; 

(3)  Providing  secure  communications  linking 
DoD  agencies  to  each  other  and  to  their 
contractors. 

Such  protection  is  obviously  costly  and  will  re¬ 
quire  additional  planning  time,  test  resources,  and 
test  constraints.  The  test  planner  must  determine 
all  possible  ways  in  which  the  system  could  be 
susceptible  to  hostile  exploitation  during  testing. 
For  example,  announcement  of  test  schedule  and 
location  could  allow  monitoring  by  unauthorized 
persons.  Knowledge  of  the  locations  of  systems 
and  instrumentation  or  test  concepts  could  reveal 
classified  system  capabilities  or  military  concepts. 
Compilations  of  unclassified  data  could,  as  a 
whole,  reveal  classified  information  as  could  sur¬ 
veillance  (electronic  or  photographic)  of  test  ac¬ 
tivities  or  interception  of  unencrypted  transmis¬ 
sions.  The  T&E  regulations  of  each  Service  re¬ 
quire  an  operational  security  plan  for  a  test.  A 
detailed  list  of  questions  the  test  planner  can  use 


to  identify  the  potential  threat  of  exploitation  is 
provided  in  AFR  55-43,  Management  of  Opera¬ 
tional  Test  and  Evaluation. 

24.5  ADVANCED  CONCEPT 
TECHNOLOGY  DEMONSTRATIONS 
(ACTDS) 

Systems  with  potential  utility  for  the  user  and  hav¬ 
ing  relatively  mature  technology  may  be  evalu¬ 
ated  by  a  user  in  an  operational  field  environment. 
These  programs  are  not  an  acquisition  program 
and  therefore  are  not  subject  to  the  normal  acqui¬ 
sition  T&E  processes.  A  favorable  evaluation  may 
result  in  the  decision  to  acquire  additional  systems 
for  Service  use,  bypassing  a  number  of  the  nor¬ 
mal  acquisition  phases.  The  Services  have  been 
using  their  operational  test  agencies  to  assist  the 
field  commanders  in  structuring  an  evaluation  pro¬ 
cess  which  would  provide  the  documented  data 
necessary  for  an  informed  acquisition  decision. 

24.6  SUMMARY 

All  weapon  systems  tests  are  limited  to  some  de¬ 
gree,  but  certain  systems  face  major  limitations  that 
could  preclude  a  comprehensive  and  realistic 
evaluation.  The  test  planners  of  these  special  sys¬ 
tems  must  allow  additional  planning  time,  budget 
for  extra  test  resources,  and  devise  alternative  test 
strategies  to  work  around  testing  limitations  caused 
by  such  factors  as  security  restrictions,  resource 
availability,  environmental  safety  factors,  and  non¬ 
standard  acquisition  strategies. 
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BEST  PRACTICES  IN  T&E 
OF  WEAPON  SYSTEMS 


25.1  INTRODUCTION 

Numerous  pre-millennium  2000  studies  were  con¬ 
ducted  by  various  agencies  that  highlighted  dif¬ 
ferent  perspectives  on  best  practices  for  Test  and 
Evaluation  (T&E).  In  June  1999,  the  Office  of 
the  Secretary  of  Defense  (OSD)  published  their 
study  conducted  by  the  Science  Applications  In¬ 
ternational  Corporation,  Best  Practices  Applicable 
to  DoD  [Department  of  Defense]  Developmental 
Test  and  Evaluation.  The  Executive  Summary 
stated  “While  the  study  team  found  no  ‘Silver 
Bullets,’  it  did  identify  some  twenty  practices  used 
by  commercial  enterprises  that  are  relevant  to 
ODTSE&E  business  practices.  These  practices 
have  been  grouped  under  the  categories  ‘Policy,’ 
‘Planning,’  ‘Test  Conduct,’  and  ‘Test  Analysis’.” 

Shortly  thereafter  in  September,  1999,  the  Defense 
Science  Board  (DSB)  Task  Force  released  its  re¬ 
port  on  a  broad  review  of  the  entire  range  of  ac¬ 
tivities  relating  to  T&E.  Their  summary  recom¬ 
mendations  were:  start  T&E  early- very  early; 
make  T&E  part  of  the  acquisition  process  —  not 
adversarial  to  it;  consolidate  Development  Test 
(DT)  and  Operational  Test  (OT);  provide  joint  test 
leadership;  fund  Modeling  and  Simulation  (M&S) 
support  of  T&E  in  program  budgets;  maintain  in¬ 
dependence  of  evaluation  process  while  integrat¬ 
ing  all  other  activities;  and,  establish  range  own¬ 
ership  and  operation  structure  separate  from  the 
Service  DT/OT  organizations.  A  follow-on  DSB 
report,  Test  and  Evaluation  Capabilities,  Decem¬ 
ber  2000,  addressed  the  value  of  testing,  manage¬ 
ment  of  T&E  resources,  quality  of  testing,  spe¬ 
cific  T&E  investments,  and  the  use  of  training 
facilities/exercises  for  T&E  events. 


In  the  same  time  frame,  A.  Lee  Battershell  released 
her  study  for  the  National  Defense  University 
(NDU)  comparing  the  acquisition  practices  of  the 
Boeing  777  and  the  Air  Force  C-17.  Her  most 
interesting  yet  not  surprising  conclusion  was  that 
some  commercial  best  practices  do  not  transfer  to 
government. 

This  was  followed  by  the  publication  of  the  GAO 
Report  Best  Practices:  A  More  Constructive  Test 
Approach  is  Key  to  Better  Weapon  System  Out¬ 
comes ”  (GAO/NSIAD-OO-199)  in  July  2000.  Af¬ 
ter  comparing  commercial  and  defense  system  de¬ 
velopment  practices  the  three  main  findings  were: 
problems  found  late  in  development  signal  weak¬ 
ness  in  testing  and  evaluation;  testing  early  to  vali¬ 
date  product  knowledge  is  a  best  practice;  and, 
different  incentives  make  testing  a  more  construc¬ 
tive  factor  in  commercial  programs  than  in  weapon 
system  programs. 

The  following  information  offers  guidance  to 
Department  of  Defense  (DoD)  personnel  who 
plan,  monitor  and  execute  T&E.  Checklists  found 
in  the  remainder  of  the  chapter  were  obtained  from 
the  DSB  Study,  Report  of  Task  Force  on  Test  and 
Evaluation,  2  April  1974.  This  excellent  study  is 
highly  regarded  in  the  T&E  community  but  has 
become  somewhat  dated;  consequently,  the  De¬ 
fense  Acquisition  University  (DAU)  decided  to 
update  the  study  findings  and  include  those  find¬ 
ings  and  summary  checklists  in  this  guide.  This 
discussion  follows  the  system  from  early  concept 
through  the  various  stages  of  technology  matura¬ 
tion  demonstrated  in  different  system/test  article 
configurations. 


25.2  SPECIFIC  WEAPON  SYSTEMS 
TESTING  CHECKLIST 

The  DSB  report  is  the  result  of  the  study  of  past 
major  weapon  systems  acquisitions.  It  was  hoped 
that  this  study  would  enhance  the  testing 
community’s  understanding  of  the  role  that  T&E 
has  had  in  identifying  system  problems  during 
the  acquisition  process.  In  the  foreword  of  the 
DSB  study,  the  authors  made  this  statement 
about  including  the  obvious  testing  activity  in 
their  checklist: 

The  T&E  expert  in  reading  this  volume 
will  find  many  precepts  which  will  strike 
him  as  of  this  type.  These  items  are  in¬ 
cluded  because  examples  were  found 
where  even  the  obvious  has  been  ne¬ 
glected,  not  because  of  incompetence  or 
lack  of  personal  dedication  by  the  people 
in  charge  of  the  program,  but  because  of 
financial  and  temporal  pressures  which 
forced  competent  managers  to  compro¬ 
mise  on  their  principles.  It  is  hoped  that 
the  inclusion  of  the  obvious  will  prevent 
repetition  of  the  serious  errors  which  have 
been  made  in  the  past  when  such  politi¬ 
cal,  economical,  and  temporal  pressures 
have  forced  project  managers  to  depart 
from  the  rules  of  sound  engineering 
practices.... In  the  long  run,  taking  short 
cuts  during  T&E  to  save  time  and  money 
will  result  in  significant  increases  in  the 
overall  costs  of  the  programs  and  in  a 
delay  of  delivery  of  the  corresponding 
weapon  systems  to  combatant  forces. 

25.2.1  Aircraft  Systems 

25.2.1.1  Concept  Assessment 

•  Test  Program/Total  Costs.  Prior  to  program 
initiation,  all  phases  of  the  aircraft  test  program 
should  be  considered  so  the  total  costs  and  the 
development  schedules  include  consideration  of 
all  likely  activities  in  the  overall  program. 


•  Test  Facilities  and  Instrumentation.  The  test 
facilities  and  instrumentation  requirements  to 
conduct  tests  should  be  generally  identified 
along  with  a  tentative  schedule  of  test 
activities. 

•  Test  Resources  and  Failures.  Ensure  that  there 
are  adequate  funds,  reasonable  amounts  of  time, 
and  acceptable  numbers  of  aircraft  planned  for 
the  various  test  program  phases,  and  that  pro¬ 
visions  are  made  for  the  occurrence  of  failures. 

•  System  Interfaces.  Consider  all  aircraft  system 
interfaces,  their  test  requirements,  and  probable 
costs  at  the  outset  of  the  concept  assessment. 

•  Major  Weapon  Subsystems.  If  the  aircraft  sys¬ 
tem  relies  on  the  successful  development  of  a 
specific  and  separately-funded  major  weapon 
(such  as  a  gun  or  missile)  in  order  to  accom¬ 
plish  its  primary  mission,  this  major  subsystem 
should  be  developed  and  tested  concurrently 
with,  or  prior  to,  the  aircraft. 

•  Propulsion  System.  If  the  aircraft  program  is 
paced  by  the  propulsion  system  development, 
an  early  advanced-development  project  for  the 
propulsion  may  be  appropriate  for  a  new 
concept. 

•  Operational  Scenario.  A  conceptual  operational 
scenario  for  operation  and  use  of  the  aircraft 
should  be  developed  so  that  general  test  plans 
can  be  designed.  This  should  include  purpose, 
roles  and  missions,  threats,  operating  environ¬ 
ments,  logistics  and  maintenance,  and  basing 
characteristics.  The  potential  range  of  values 
on  these  aspects  should  be  stated. 

•  Evaluation  Criteria.  Develop  evaluation  crite¬ 
ria  to  be  used  for  selecting  the  final  aircraft 
system  design. 

•  Untried  Elements.  The  aircraft  development 
program  should  include  conclusive  testing  to 
eliminate  uncertainties  of  the  untried  elements. 
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•  Brassboard  Avionics  Tests.  The  use  of  brass- 
board  or  modified  existing  hardware  to  “prove” 
the  concept  will  work  should  be  seriously  scru¬ 
tinized  to  ensure  that  the  demonstrations  and 
tests  are  applicable. 

•  Nuclear  Weapons  Effects.  The  subject  of 
nuclear  weapons  effects  should  be  addressed 
in  the  test  concept  for  all  aircraft  weapons  sys¬ 
tems  where  operational  suitability  dictates  that 
survivable  exposure  to  nuclear  weapons  effects 
is  a  requirement. 

25.2.1.2  Prototype  Development 

•  T&E  Strategy.  T&E  plans  and  test  criteria 
should  be  established  so  there  is  no  question 
on  what  constitutes  a  successful  test  and  what 
performance  is  required. 

•  Milestones  and  Goals.  Ensure  an  integrated 
system  test  plan  that  pre-establishes  milestones 
and  goals  for  easy  measurement  of  program 
progress  at  a  later  time. 

•  Operating  Concept  and  Environment.  The  op¬ 
erational  concept  and  the  environments  in 
which  the  aircraft  will  be  expected  to  operate 
and  be  tested  in  Operational  Test  and  Evalua¬ 
tion  (OT&E)  should  be  specified. 

•  Test  Program  Building  Blocks.  In  testing  the 
prototype,  demonstrate  that  high-risk  technol¬ 
ogy  is  in  hand.  In  planning  the  system  level 
test  program,  ensure  that  components  and  sub¬ 
systems  are  adequately  qualified  for  incorpora¬ 
tion  into  the  system  tests. 

•  Technology  Concepts.  Each  concept  to  be  used 
in  the  aircraft  system  (e.g.,  aerodynamics, 
structures,  propulsion)  should  be  identified  and 
coded  according  to  prior  application,  before 
future  research.  Tests  for  each  concept  should 
be  specified  with  the  effect  of  failure 
identified. 


•  Development  Test  and  Evaluation  ( DT&E ')/ 
OT&E  Plan.  The  aircraft  DT&E/OT&E  test 
plan  should  be  reviewed  to  ensure  it  includes 
ground  and  flight  tests  necessary  to  safely  and 
effectively  develop  the  system. 

•  Test  Failures.  The  T&E  plans  should  be  made 
assuming  there  will  be  failures;  they  are 
inevitable. 

•  Multi-Service  Testing.  When  a  new  aircraft  de¬ 
velopment  program  requires  multi-Service  test¬ 
ing  during  OT&E  and  prior  to  Low  Rate  Ini¬ 
tial  Production  (LRIP),  the  test  plan  should  in¬ 
clude  the  types  of  tests  and  resources  required 
from  other  activities  and  Services. 

•  Traceability.  The  aircraft  development  and  test 
program  should  be  designed  and  scheduled  so 
if  trouble  arises,  its  source  can  be  traced  back 
through  the  lab  tests  and  the  analytical  studies. 

•  Competitive  Prototype  Tests.  When  a  competi¬ 
tive  prototype  test  program  using  test  and  op¬ 
erational  crews  is  employed,  the  aircraft  should 
be  compared  on  the  basis  of  the  performance 
of  critical  missions. 

•  Prototype  Similarity  to  Development  and  Pro¬ 
duction  Aircraft.  A  firm  determination  should 
be  made  of  the  degree  of  similarity  of  the  win¬ 
ning  prototype  (in  a  competitive  prototype  pro¬ 
gram)  to  the  Engineering  Development  Model 
(EDM)  and  production  aircraft.  Thus,  test  re¬ 
sults  that  are  derived  from  the  prototype  in  the 
interim  period  prior  to  availability  of  the  EDM 
aircraft  can  be  utilized  effectively. 

•  Prototype  Tests.  The  prototype  aircraft  test 
data  should  be  used  to  determine  where  em¬ 
phasis  should  be  placed  in  the  engineering 
development  program. 

•  Inlet/Engine/Nozzle  Match.  The  aircraft  test 
program  should  provide  for  an  early  and  ad¬ 
equate  inlet/engine/nozzle  match  through  a 
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well-planned  test  program,  and  there  should  be 
time  programming  for  corrections. 

•  Subsystem  Tests.  There  should  be  a  balanced 
program  for  the  aircraft  subsystem  tests. 

•  Propulsion  System.  If  the  aircraft  is  paced  by 
the  propulsion  systems  development,  an  early 
advanced-development  project  for  the  propul¬ 
sion  may  be  appropriate  for  a  new  concept. 

•  Electromagnetic  Interface  (EMI)  Testing.  Full- 
scale  aircraft  systems  tests  in  an  anechoic  cham¬ 
ber  are  desirable  for  some  aircraft. 

•  Parts  Interchange.  Early  plans  should  provide 
for  tests  where  theoretically  identical  parts,  par¬ 
ticularly  in  avionics,  are  interchanged  to  ensure 
that  the  aircraft  systems  can  be  maintained  in 
readiness. 

•  Human  Factors  Demonstration.  Ensure  ad¬ 
equate  demonstration  of  human  factors  is  con¬ 
sidered  in  test  planning. 

•  Logistics  T&E.  Adequate  resources  should  be 
scheduled  for  the  aircraft  logistics  system  T&E 
and  a  positive  program  should  exist  for  the 
utilization  of  this  information  at  the  time  of 
OT&E. 

•  User  Participation.  It  is  imperative  that  the 
operational  command  actively  participate  in  the 
DT&E  phase  to  ensure  that  user  needs  are  rep¬ 
resented  in  the  development  of  the  system. 

25.2.1.3  Engineering  Development  Model 

•  Test  Design.  Test  programs  should  be  designed 
to  have  a  high  probability  of  early  identifica¬ 
tion  of  major  deficiencies  during  the  DT&E 
and  Initial  Operational  Test  and  Evaluation 
(IOT&E). 

•  Data  for  Alternate  Scenarios.  By  careful  at¬ 
tention  to  testing  techniques,  maximize  the 


utility  of  the  test  data  gathered;  aircraft 
instrumentation;  range  instrumentation;  and  data 
collection,  reduction,  and  storage. 

•  Test  Milestones.  Development  programs  should 
be  built  around  testing  milestones,  not  calendar 
dates. 

•  Production  Engineering  Influence  on  Research 
and  Development  (R&D)  Hardware.  Encour¬ 
age  that  production  philosophy  and  production 
techniques  be  brought  to  the  maximum  practi¬ 
cable  extent  into  an  early  phase  of  the  design 
process  for  R&D  hardware. 

•  Running  Evaluation  of  Tests.  Ensure  that  run¬ 
ning  evaluations  of  tests  are  conducted.  If  it 
becomes  clear  that  test  objectives  are  unattain¬ 
able  or  additional  samples  will  not  change  the 
test  outcome,  ensure  that  procedures  are 
established  for  terminating  the  test. 

•  Simulation.  Analysis  and  simulation  should  be 
conducted,  where  practicable,  before  each 
phase  of  development  flight  testing. 

•  Avionics  Mock-up.  Encourage  use  of  a  com¬ 
plete  avionics  system  installed  in  a  mock-up  of 
the  appropriate  section  or  sections  of  the  aircraft. 

•  Escape  Systems  Testing.  Ensure  the  aircrew 
escape  system  is  thoroughly  tested  with  par¬ 
ticular  attention  to  redundant  features,  such  as 
pyrotechnic  firing  channels. 

•  Structural  Testing.  Ensure  that  fatigue  testing  is 
conducted  on  early  production  airframes.  Air¬ 
frame  production  should  be  held  to  a  low  rate 
until  satisfactory  progress  is  shown  in  these  tests. 

•  Gun  Firing  Tests.  All  forms  of  ordnance,  espe¬ 
cially  those  that  create  gases,  must  be  fired  from 
the  aircraft  for  external  effects  (blast  and  de¬ 
bris),  internal  effects  (shock)  and  effects  on  the 
propulsion  (inlet  composition  or  distribution). 
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•  Post-Stall  Characteristics.  Special  attention  is 
warranted  on  the  post-stall  test  plans  for  DT&E 
and  OT&E. 

•  Subsystem  Performance  History.  During  DT&E 
and  IOT&E  of  aircraft,  ensure  that  a  performance 
history  of  each  aircraft  subsystem  is  kept. 

•  Flight  Deficiency  Reporting.  Composition  of 
flight  deficiencies  reporting  by  aircrews,  par¬ 
ticularly  those  pertaining  to  avionics,  should  be 
given  special  attention. 

•  Crew  Limitations.  Ensure  aircrew  limitations 
are  included  in  the  tests. 

•  Use  of  Operational  Personnel.  Recommend  ex¬ 
perienced  operational  personnel  help  in  estab¬ 
lishing  Measures  of  Effectiveness  (MOEs)  and 
in  other  operational  test  planning.  In  conduct¬ 
ing  OT&E,  use  typical  operational  aircrews  and 
support  personnel. 

•  Role  of  the  User.  Ensure  that  users  participate 
in  the  T&E  phase  so  their  needs  are  represented 
in  the  development  of  the  system  concept  and 
hardware. 

•  Crew  Fatigue  and  System  Effectiveness.  In  at¬ 
tack  aircraft  operational  testing  and  particularly 
in  attack  helicopter  tests  where  vibration  is  a 
fatiguing  factor,  ascertain  that  the  tests  include 
a  measure  of  degradation  over  time. 

•  Time  Constraints  on  Crews.  Detailed  OT  plans 
should  be  evaluated  to  determine  that  the  test- 
imposed  conditions  on  the  crew  do  not  invali¬ 
date  the  applicability  of  the  collected  data. 

•  Maintenance  and  Training  Publications.  The 
aircraft  development  program  should  provide 
for  concurrent  training  of  crews  and  prepara¬ 
tion  of  draft  technical  manuals  to  be  used  by 
IOT&E  maintenance  and  operating  crews. 


•  Research  and  Development  (R&D)  Completion 
Prior  to  IOT&E.  The  testing  plans  should  en¬ 
sure  that,  before  an  aircraft  system  is  subjected 
to  IOT&E,  the  subsystems  essential  to  the  basic 
mission  have  completed  R&D. 

•  Complete  Basic  DT&E  before  Starting  OT&E. 
Before  the  weapon  system  is  subjected  to 
IOT&E,  all  critical  subsystems  should  have 
completed  basic  DT&E  and  significant 
problems  should  be  solved. 

•  Realism  in  Testing.  Ascertain  that  final  DT&E 
system  tests  and  IOT&E  flight  tests  are 
representative  of  operational  conditions. 

•  Test  All  Profiles  and  Modes.  Tests  should  be 
conducted  to  evaluate  all  planned  operational 
flight  profiles  and  all  primary  and  backup, 
degraded  operating  modes. 

•  Update  of  OT  Plans.  Ensure  that  OT  plans  are 
reviewed  and  updated,  as  needed,  to  make  them 
relevant  to  evolving  concepts. 

•  Plan  OT&E  Early.  Ensure  that  operational  suit¬ 
ability  tests  are  planned  to  attempt  to  identify 
operational  deficiencies  of  new  systems  quickly 
so  fixes  can  be  developed  and  tested  before 
large-scale  production. 

•  Missile  Launch  Tests.  Review  the  final  posi¬ 
tion  fix  planned  before  launching  inertial-guided 
air-to-surface  missiles. 

•  Mission  Completion  Success  Probability.  Mis¬ 
sion  completion  success  probability  factors 
should  be  used  to  measure  progress  in  the 
aircraft  test  program. 


25-5 


25.2.1.4  Production  (LRIP  and  Full  Rate), 
Deployment  and  Operational 
Support 

•  OT  Realism.  Assure  IOT&E  and  Follow-on 
Test  and  Evaluation  (FOT&E)  are  conducted 
under  realistic  conditions. 

•  Design  FOT&E  for  Less-Than-Optimal  Con¬ 
dition.  Structure  the  FOT&E  logistical  support 
for  simulated  combat  conditions. 

•  New  Threat.  Be  alert  to  the  need  to  extend  the 
IOT&E  if  a  new  threat  shows  up.  Address 
IOT&E  limitations  in  FOT&E. 

•  Certification  of  Ordnance.  Ensure  that  ordnance 
to  be  delivered  by  an  aircraft  is  certified  for  the 
aircraft. 

•  Inadvertent  Influence  of  Test.  The  IOT&E/ 
FOT&E  plans  should  provide  measures  for 
ensuring  that  actions  by  observers  and  umpires 
do  not  unwittingly  influence  trial  outcome. 

•  Deficiencies  Discovered  In-Service.  Be  aware 
that  in-Service  operations  of  an  aircraft  system 
will  surface  deficiencies  which  extensive 
IOT&E/FOT&E  probably  would  not  uncover. 

•  Lead  the  Fleet.  Accelerated  Service  test  of  a 
small  quantity  of  early  production  aircraft  is 
advisable  and  periodically  during  FOT&E 
thereafter. 

25.2.2  Missile  Systems 

25.2.2.1  Concept  Assessment 

•  Weapon  System  Interfaces .  Consider  significant 
weapon  system  interfaces,  their  test  require¬ 
ments  and  probable  costs  at  the  outset  of  the 
concept  assessment.  Ensure  that  the  program 
plan  assembled  before  program  start  includes 
an  understanding  of  the  basic  test  criteria  and 
broad  test  plans  for  the  whole  program. 


•  Number  of  Test  Missiles.  Ensure  that  there  is 
sufficient  time  and  a  sufficient  number  of  test 
articles  to  support  the  program  through  its  vari¬ 
ous  phases.  Compare  the  program  requirements 
with  past  missile  programs  of  generic  similar¬ 
ity.  If  there  is  substantial  difference,  then  ad¬ 
equate  justification  should  be  provided.  The 
DT&E  period  on  many  programs  has  had  to 
be  extended  as  much  as  50  percent. 

•  T&E  Gap.  A  T&E  gap  has  been  experienced 
in  some  missile  programs  between  the  time 
when  testing  with  R&D  hardware  was  com¬ 
pleted  and  the  time  when  follow-on  opera¬ 
tional  suitability  testing  was  initiated  with  pro¬ 
duction  hardware. 

•  Feasibility  Tests.  Ensure  experimental  test  evi¬ 
dence  is  available  to  indicate  the  feasibility  of 
the  concept  and  the  availability  of  the  technol¬ 
ogy  for  the  system  development. 

•  Evaluation  of  Component  Tests.  Results  of  tests 
conducted  during  the  concept  assessment  and 
the  prototype  testing,  which  most  likely  have 
been  conducted  as  avionics  brassboard,  bread¬ 
board,  or  modified  existing  hardware,  should 
be  evaluated  with  special  attention. 

•  Multi-Service  Testing  Plans.  When  a  new  mis¬ 
sile  development  program  requires  multi- 
Service  testing  during  OT&E,  the  early  Test 
and  Evaluation  Master  Plan  (TEMP)  should  in¬ 
clude  the  type  of  tests  and  resources  required 
from  other  activities  and  Services. 

•  Test  Facilities  and  Instrumentation  Require¬ 
ments.  The  test  facilities  and  instrumentation 
requirements  to  conduct  tests  should  be  gener¬ 
ally  identified  early  along  with  a  tentative 
schedule  of  test  activities. 

25.2.2.2  Prototype  Testing 

•  Establish  Test  Criteria.  By  the  end  of  prototype 
testing,  test  criteria  should  be  established  so  there 
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is  no  question  on  what  constitutes  a  successful 
test  and  what  performance  is  expected. 

•  Human  Factors.  Ensure  that  the  TEMP  in¬ 
cludes  adequate  demonstration  of  human  fac¬ 
tors  considerations. 

•  Instrumentation  Diagnostic  Capability  and 
Compatibility.  Instrumentation  design,  with 
adequate  diagnostic  capability  and  compatibil¬ 
ity  in  DT&E  and  IOT&E  phases,  is  essential. 

•  Provisions  for  Test  Failures.  The  DT&E  and 
OT&E  plans  should  include  provisions  for  the 
occurrence  of  failures. 

•  Integrated  Test  Plan  (ITP).  Ensure  development 
of  an  integrated  system  test  plan  that  pre-estab¬ 
lishes  milestones  and  goals  for  easy  measure¬ 
ment  of  program  progress  at  a  later  time. 

•  T&E  Requirements.  Ensure  that  the  T&E  pro¬ 
gram  requirements  are  firm  before  approving 
an  R&D  test  program.  Many  missile  programs 
have  suffered  severe  cost  impacts  as  a  result  of 
this  deficiency.  The  test  plan  must  include  pro¬ 
visions  to  adequately  test  those  portions  of  the 
operational  envelope  that  stress  the  system  in¬ 
cluding  backup  and  degraded  operational 
modes. 

•  Personnel  Training  Plans.  Ensure  that  adequate 
training  and  certification  plans  for  test  person¬ 
nel  have  been  developed. 

•  Test  and  Engineering  Reporting  Format.  In¬ 
clude  a  T&E  reporting  format  in  the  program 
plan.  Attention  must  be  given  to  the  reporting 
format  in  order  to  provide  a  consistent  basis 
for  T&E  throughout  the  program  life  cycle. 

•  Program-to-Program  Cross  Talk.  Encourage  pro- 
gram-to-program  T&E  cross  talk.  T&E  problems 
and  their  solutions,  as  one  program,  provide  a 
valuable  index  of  lessons  learned  and  techniques 
for  problem  resolution  on  other  programs. 


•  Status  of  T&E  Offices.  Ensure  that  T&E  of¬ 
fices  reporting  to  the  program  manager  or  di¬ 
rector  have  the  same  stature  as  other  major  el¬ 
ements.  It  is  important  that  the  T&E  compo¬ 
nent  of  the  system  program  office  has  organi¬ 
zational  status  and  authority  equal  to  configu¬ 
ration  management,  program  control,  system 
engineering,  etc. 

•  Measurement  of  Actual  Environments.  Thor¬ 
ough  measurements  should  be  made  to  define 
and  understand  the  actual  environment  in  which 
the  system  components  must  live  during  the 
captive,  launch,  and  in-flight  phases. 

•  Thoroughness  of  Laboratory  Testing.  Significant 
time  and  money  will  be  saved  if  each  compo¬ 
nent,  each  subsystem,  and  the  full  system  are  all 
tested  as  thoroughly  as  possible  in  the  laboratory. 

•  Contract  Form.  The  contract  form  can  be  ex¬ 
tremely  important  to  the  T&E  aspects.  In  one 
program,  the  contract  gave  the  contractor  full 
authority  to  determine  the  number  of  test  mis¬ 
siles;  and  in  another,  the  contract  incentive  re¬ 
sulted  in  the  contractor  concentrating  tests  on 
one  optimum  profile  to  satisfy  the  incentive 
instead  of  developing  the  performance  through¬ 
out  important  areas  of  the  envelope. 

•  Participation  of  Operational  Command.  It  is 
imperative  that  the  operational  command  ac¬ 
tively  participate  in  the  DT&E  phase  to  ensure 
that  user  needs  are  represented  in  the  develop¬ 
ment  of  the  system. 

25.2.2.3  Engineering  Development  Model 

•  Production  Philosophy  and  Techniques.  En¬ 
courage  that  production  philosophy  and  pro¬ 
duction  techniques  be  brought,  to  the  maximum 
practicable  extent,  into  an  early  phase  of  the 
design  process  for  R&D  hardware.  There  are 
many  missile  programs  in  which  the  compo¬ 
nents  were  not  qualified  until  the  missile  was 
well  into  production. 
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•  Operational  Flight  Profiles.  Tests  should  be 
conducted  to  evaluate  all  planned  operational 
flight  profiles  and  all  primary  and  backup 
degraded  operating  modes. 

•  Failure  Isolation  and  Responsive  Action.  Does 
the  system  test  plan  provide  for  adequate  in¬ 
strumentation  so  missile  failures  can  be  isolated 
and  fixed  before  the  next  flight? 

•  Responsive  Actions  for  Test  Failures.  Encour¬ 
age  a  closed-loop  reporting  and  resolution  pro¬ 
cess,  which  ensures  that  each  test  failure  at 
every  level  is  closed  out  by  appropriate  action; 
i.e.,  redesign,  procurement,  retest,  etc. 

•  Plan  Tests  of  Whole  System.  Plan  tests  of  the 
whole  system  including  proper  phasing  of  the 
platform  and  supporting  gear,  the  launcher,  the 
missile  and  user  participation. 

•  Determination  of  Component  Configuration. 
Conditions  and  component  configuration  dur¬ 
ing  development  tests  should  be  determined  by 
the  primary  objectives  of  that  test.  Whenever  a 
non-operational  configuration  is  dictated  by 
early  test  requirements,  tests  should  not  be  chal¬ 
lenged  by  the  fact  that  configuration  is  not 
operational. 

•  Testing  of  Software.  T&E  should  ensure  that 
software  products  are  tested  appropriately  dur¬ 
ing  each  phase.  Software  often  has  been  de¬ 
veloped  more  as  an  add-on  than  as  an  integral 
part  of  the  overall  system.  Software  require¬ 
ments  need  the  same  consideration  as  hardware 
requirements  in  the  prototype  development. 

•  Range  Safety  Dry  Runs.  Ensure  the  test  plan 
includes  adequate  test  program/range  safety  dry 
runs.  The  government  test  ranges  have  to  pro¬ 
vide  facilities  to  safely  test  many  different 
projects: 

—  Assemblies/Subsystems  Special  Require¬ 
ments, 


—  Seekers  and  tracking  devices, 

—  Propulsion  subsystems, 

—  Connectors  and  their  related  hardware, 

—  Lanyard  assemblies, 

—  Safing,  arming,  fuzing,  and  other  ordnance 
devices. 

•  Review  of  Air-to-Surface  Missile  (ASM)  Test 
Position  Fixes.  Review  the  final  position  fix 
planned  before  launching  ASMs.  There  are 
instances  in  which  the  OT  of  air-launched  mis¬ 
siles  utilized  artificial  position  fixes  just  prior 
to  missile  launch. 

•  Operator  Limitations.  Ensure  operator  limita¬ 
tions  are  included  in  the  tests.  Most  tactical 
missiles,  especially  those  used  in  close  support, 
require  visual  acquisition  of  the  target  by  the 
missile  operator  and/or  an  air/ground  controller. 

•  Test  Simulations  and  Dry  Runs.  Plan  and  use 
test  simulations  and  dry  runs.  Dry  runs  should 
be  conducted  for  each  new  phase  of  testing. 
Simulation  and  other  laboratory  or  ground  test¬ 
ing  should  be  conducted  to  predict  the  specific 
test  outcome.  The  “wet  run”  test  should  finally 
be  run  to  verify  the  test  objectives.  Evaluation 
of  the  simulation  versus  the  actual  test  results 
will  help  to  refine  the  understanding  of  the 
system. 

•  Component  Performance  Records.  Keep  per¬ 
formance  records  on  components.  There  are 
many  examples  in  missile  programs  that  have 
required  parts  stock  sweeps  associated  with 
flight  failures  and  component  aging  test 
programs. 

•  Tracking  Test  Data.  Ensure  the  test  program 
tracks  data  in  a  readily  usable  manner.  Reli¬ 
ability  and  performance  evaluations  of  a  mis¬ 
sile  system  should  break  down  the  missile’s 
activity  into  at  least  the  following  phases: 

—  Prelaunch  including,  captive  carry  reliability, 
—  Launch, 
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—  In-flight, 

—  Accuracy/fuzing. 

•  Updating  IOT &E  Planning.  Periodically  update 
Production  Qualification  Testing  (PQT)  and 
IOT&E  planning  during  the  early  R&D  phase. 
Few  missile  system  programs  have  had  ad¬ 
equate  user  participation  with  the  desired 
continuity  of  personnel  to  minimize  the  prob¬ 
lems  of  transition  from  DT&E  to  OT&E  to 
deployment/utilization. 

•  Instrumentation  Provisions  in  Production  Mis¬ 
siles.  Encourage  built-in  instrumentation  pro¬ 
visions  in  production  missiles. 

•  Constraints  on  Missile  Operator.  Detailed  test 
plans  should  be  evaluated  to  determine  that  the 
test  imposed  constraints  on  the  missile  opera¬ 
tor  do  not  invalidate  the  applicability  of  the  data 
so  collected. 

•  Problem  Fixes  Before  Production.  Ensure  that 
operational  suitability  tests  identify  operational 
deficiencies  of  new  systems  quickly  so  fixes 
can  be  developed  and  tested  before  production. 

•  Flight  Tests  Representative  of  Operations.  As¬ 
certain  that  final  DT&E  system  tests  and 
planned  IOT&E  flight  tests  are  representative 
of  operational  flights.  Some  ballistic  missile 
R&D  programs  have  shown  high  success  rates 
in  R&D  flight  tests;  however,  when  the  early 
production  systems  were  deployed,  they  exhib¬ 
ited  a  number  of  unsatisfactory  characteristics 
such  as  poor  alert  reliability  and  poor  opera¬ 
tional  test-flight  reliability. 

•  System  Interfaces  in  OT.  Ensure  the  primary 
objective  of  an  OT  planning  is  to  obtain  mea¬ 
surements  on  the  overall  performance  of  the 
weapon  system  when  it  is  interfaced  with  those 
systems  required  to  operationally  use  the  weap¬ 
ons  system. 


25.2.2.4  Production  (LRIP,  Full  Rate), 
Deployment  and  Operational 
Support 

•  Realistic  Conditions  for  Operational  Testing. 
Ascertain  operational  testing  is  conducted  under 
realistic  combat  conditions.  This  means  that  the 
offense/defense  battle  needs  to  be  simulated  in 
some  way  before  the  weapon  system  evaluation 
can  be  considered  completed.  Whether  this 
exercise  is  conducted  within  a  single  Service 
(as  in  the  test  of  a  surface-to-surface  antitank 
missile  against  tanks)  or  among  Services  (as  in 
the  test  of  an  ASM  against  tanks  with  antiair¬ 
craft  protection),  the  plans  for  such  testing 
should  be  formulated  as  part  of  the  system  de¬ 
velopment  plan. 

•  Testing  All  Operational  Modes.  Ensure  the 
FOT&E  plan  includes  tests  of  any  operational 
modes  not  previously  tested  in  IOT&E.  All 
launch  modes  including  degraded,  backup 
modes  should  be  tested  in  the  FOT&E  because 
the  software  interface  with  the  production  hard¬ 
ware  system  should  be  evaluated  thoroughly. 
Otherwise,  small,  easy-to-fix  problems  might 
preclude  launch. 

•  Extension  of  the  OT&E  for  New  Threats.  Be 
alert  to  the  need  to  extend  the  IOT&E/FOT&E 
if  a  new  threat  arises.  Few  missile  programs 
perform  any  kind  of  testing  relatable  to  evalu¬ 
ating  system  performance  against  current  or 
new  threats. 

•  “Lead-the-Fleet”  Production  Scheduling. 
Lead-the-Fleet  missile  scheduling  and  tests 
should  be  considered. 

•  Test  Fixes.  Test  fixes  result  from  earlier  opera¬ 
tional  testing.  After  the  IOT&E  that  identified 
problem  areas  in  missiles,  FOT&E  should 
evaluate  these  areas  primarily  to  determine  the 
adequacy  of  the  incorporated  fixes,  particularly 
if  the  IOT&E  did  not  run  long  enough  to  test 
the  fixes. 
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•  FOT&E  Feedback  to  Acceptance  Testing.  En¬ 
sure  that  FOT&E  results  are  quickly  fed  back 
to  influence  early  production  acceptance  test¬ 
ing.  Production  acceptance  testing  is  probably 
the  final  means  the  government  normally  will 
have  to  ensure  the  product  meets  specifications. 
Early  acceptance  testing  could  be  influenced 
favorably  by  a  quick  feedback  from  FOT&E 
to  acceptance  testing.  This  is  exemplified  by  a 
current  ASM  program  where  production  has 
reached  peak  rates,  and  the  IOT&E  has  not 
been  completed. 

25.2.3  Command  and  Control  (C2)  Systems 

25.2.3.1  Concept  Assessment 

•  Concept  Test  Philosophy.  The  T&E  planners 
must  understand  the  nature  of  Command  and 
Control  (C2)  systems  early  in  the  concept  as¬ 
sessment.  In  a  complex  C2  system,  a  total  sys¬ 
tems  concept  must  be  developed  initially.  Total 
systems  life  cycle  must  be  analyzed  so  the  nec¬ 
essary  requirement  for  the  design  can  be 
established. 

•  The  Importance  of  Software  Testing.  Testers 
should  recognize  that  software  is  a  pacing  item 
in  C2  systems  development. 

•  Software  Test  Scheduling  —  Contractors’  Fa¬ 
cilities.  Provision  should  be  made  for  includ¬ 
ing  software  T&E  during  each  phase  of  C2  sys¬ 
tems’  acquisition.  Availability  of  contractors’ 
facilities  should  be  considered. 

•  Evaluation  of  Exploratory  Development  Tests. 
Care  should  be  exercised  in  evaluating  results 
of  tests  conducted  during  exploratory  develop¬ 
ment  of  C2  systems.  These  tests,  which  most 
likely  have  been  conducted  on  brassboard, 
breadboard,  or  modified  existing  hardware, 
should  be  evaluated  with  special  attention. 

•  Feasibility  Testing  for  Field  Compilers.  Early 
test  planning  should  allow  for  simulating  the 


computer  system  to  test  for  field  use  of 
compilers,  where  applicable. 

•  Evaluation  of  Test  Plan  Scheduling.  Milestones 
should  be  event-oriented,  not  calendar-oriented. 

•  Type  Personnel  Needs  —  Effects  on  T&E.  A 
mix  of  personnel  with  different  backgrounds 
affecting  T&E  is  required. 

•  Planning  for  Joint-Service  OT&E  Before  Pro¬ 
gram  Start.  A  Joint-Service  OT&E  (multi- 
Service)  T&E  strategy  should  be  considered  for 
C2  systems. 

25.2.3.2  Prototype  Testing 

•  Test  Prototypes.  In  C2  systems,  prototypes  must 
reasonably  resemble  final  hardware  configura¬ 
tion  from  a  functional-use  standpoint.  When 
high  technical  risk  is  present,  development 
should  be  structured  around  the  use  of  one  or 
more  test  prototypes  designed  to  prove  the  sys¬ 
tem  concept  under  realistic  operational  condi¬ 
tions  before  proceeding  to  engineering 
development. 

•  Test  Objectives  —  Critical  Issues.  In  addition 
to  addressing  critical  technical  issues,  T&E 
objectives  during  prototype  testing  should  ad¬ 
dress  the  functional  issues  of  a  C2  system. 

•  Real-Time  Software  —  Demonstration  of  “Ap¬ 
plication  Patches.”  Tests  of  real-time  C2  sys¬ 
tems  should  include  demonstrations  of  inter¬ 
faces  whereby  locally  generated  application 
patches  are  brought  into  being. 

•  Independent  Software  Test-User  Group.  An 
independent  test-user  software  group  is  needed 
during  early  software  qualification  testing. 

•  System  Interfaces.  Critical  attention  should  be 
devoted  to  testing  interfaces  with  other  C2  sys¬ 
tems  and  to  interfaces  between  subsystems. 
Particular  attention  should  be  devoted  to 
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interfaces  with  other  C2  systems  and  to  the  in¬ 
terfaces  between  sensors  (e.g.,  radar  units), 
communications  systems  (e.g.,  modems)  and 
the  specific  processors  (e.g..  Central  Process¬ 
ing  Units  (CPUs)).  Interface  with  information 
processing  C2  systems  must  also  address  data- 
element  and  code-standardization  problems  if 
data  is  to  be  processed  online. 

•  Human  Factors.  In  a  C2  system,  human  fac¬ 
tors  must  be  considered  from  the  earliest  pro¬ 
totype  designs  and  testing  provided.  Testing 
should  be  conducted  to  determine  the  most  ef¬ 
ficient  arrangement  of  equipment  from  the  hu¬ 
man  factor  standpoint;  e.g.,  displays  should  be 
arranged  for  viewing  from  an  optimum  angle 
whenever  possible;  adequate  maneuvering  room 
within  installation  constraints  should  be  allowed 
considering  the  number  of  personnel  normally 
manning  the  facility;  and  console-mounted  con¬ 
trols  should  be  designed  and  located  to  facilitate 
operation,  minimize  fatigue,  and  avoid  confusion. 

•  Degraded  Operations  Testing.  When  the  ex¬ 
pected  operational  environment  of  a  C2  system 
suggests  that  the  system  may  be  operated  un¬ 
der  less-than-finely-tuned  conditions,  tests 
should  be  designed  to  allow  for  performance 
measurements  under  degraded  conditions. 

•  Test-Bed.  The  use  of  a  test-bed  for  study  and 
experimentation  with  new  C2  systems  is  needed 
early  in  the  prototype  development. 

•  Software-Hardware  Interfaces.  The  software- 
hardware  interfaces,  with  all  operational  backup 
modes  to  a  new  C2  system,  should  be  tested 
early  in  the  program. 

•  Reproducible  Tests.  Test  plans  should  contain 
a  method  for  allowing  full-load  message  input 
while  maintaining  reproducible  test  conditions. 

•  Cost-Effectiveness.  Field-test  data  is  needed 
during  the  prototype  development  for  input  to 
cost-effectiveness  analyses  of  C2  systems. 


25.2.3.3  Engineering  Development  Model 

•  Acquisition  Strategy.  The  acquisition  strategy 
for  the  system  should: 

—  Allow  sufficient  time  between  the  planned 
end  of  demonstration  testing  and  major  pro¬ 
curement  (as  opposed  to  limited  procure¬ 
ment)  decisions.  This  provides  flexibility  for 
modifying  plans,  which  may  be  required 
during  the  test  phases  of  the  program.  For 
instance,  because  insufficient  time  was  al¬ 
lowed  for  testing  one  recent  C2  system,  the 
program  and  the  contract  had  to  be  modi¬ 
fied  and  renegotiated; 

—  Be  evaluated  relative  to  constraints  im¬ 
posed; 

—  Ensure  that  sufficient  dollars  are  available, 
not  only  to  conduct  the  planned  T&E  but 
to  allow  for  the  additional  T&E  that  is  al¬ 
ways  required  due  to  failures,  design 
changes,  etc. 

•  Problem  Indications.  It  is  important  to  estab¬ 
lish  an  early  detection  scheme  so  management 
can  determine  when  a  program  is  becoming 
“ill.” 

•  Impact  of  Software  Failures.  Prior  to  any  pro¬ 
duction  release,  the  impact  of  software  failures 
on  overall  system  performance  parameters  must 
be  considered. 

•  Displays.  The  display  subsystems  of  a  C2  sys¬ 
tem  should  provide  an  essential  function  to  the 
user.  Displays  are  key  subsystems  of  a  C2  sys¬ 
tem.  They  provide  the  link  that  couples  the 
operator  to  the  rest  of  the  system  and  are,  there¬ 
fore,  often  critical  to  its  success. 

•  Pilot  Test.  A  pilot  test  should  be  conducted 
before  IOT&E  so  sufficient  time  is  available 
for  necessary  changes. 

•  Publications  and  Manuals.  It  is  imperative  that 
all  system  publications  and  manuals  be 
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completed,  reviewed,  and  selectively  tested 
under  operational  conditions  before  beginning 
overall  system  suitability  testing. 

•  Power  Sources.  Mobile,  prime  power  sources 
are  usually  provided  as  Government-Furnished 
Equipment  (GFE)  and  can  be  a  problem  area 
in  testing  C2  systems. 

•  Subsystem  Tests.  Every  major  subsystem  of  a  C2 
system  should  have  a  successful  DT&E  before 
beginning  overall  system  operational  testing. 

•  Communications.  The  C2  systems  must  be 
tested  in  the  appropriate  electromagnetic  envi¬ 
ronment  to  determine  the  performance  of  its 
communications  system. 

•  Demonstration  of  Procedures.  Test  plans  should 
include  a  procedural  demonstration  whereby 
the  tested  C2  system  works  in  conjunction  with 
other  systems. 

•  GFE  and  Facilities.  T&E  should  be  concerned 
about  the  availability  of  GFE  as  specified  in 
the  proposed  contract. 

•  User  Participation  in  T&E.  The  varying  needs 
of  the  user  for  a  C2  system  make  participation 
in  all  phases  of  T&E  mandatory. 

25.2.3.4  Production  (LRIP,  Full  Rate), 
Deployment  and  Operational 
Support 

•  Critical  Issues.  IOT&E  should  be  designed 
during  early  planning  to  provide  the  answers 
to  some  critical  issues  peculiar  to  C2  systems. 
Some  of  these  critical  issues  that  IOT&E  of  C2 
systems  should  be  planned  to  answer  are: 

—  Is  system  mission  reaction  time  a  signifi¬ 
cant  improvement  over  present  systems? 
—  Is  a  backup  mode  provided  for  use  when 
either  airborne  or  ground  system  exhibits  a 
failure? 


—  Can  the  system  be  transported  as  opera¬ 
tionally  required  by  organic  transport? 
(Consider  ground,  air,  and  amphibious 
requirements.) 

—  Is  there  a  special  requirement  for  site  prepa¬ 
ration?  (For  example,  survey  and  antenna 
location.) 

—  Can  the  system  be  erected  and  dismantled 
in  times  specified?  Are  these  times  realistic? 
—  Does  relocation  affect  system  alignment? 
—  Does  system  provide  for  operation  during 
maintenance? 

—  Can  on-site  maintenance  be  performed  on 
shelterless  subsystems  (e.g.,  radar  units) 
during  adverse  weather  conditions? 

•  IOT&E  Reliability  Data.  The  IOT&E  can  pro¬ 
vide  valuable  data  on  the  operational  reliability 
of  a  C2  system;  this  data  cannot  be  obtained 
through  DT&E. 

•  Maintenance.  In  IOT&E,  maintenance  should 
include:  a  measurement  of  the  adequacy  of  the 
maintenance  levels  and  the  maintenance  prac¬ 
tices;  an  assessment  of  the  impact  that  the  main¬ 
tenance  plan  has  on  the  operational  reliability; 
the  accessibility  of  the  major  components  of 
the  system  for  field  maintenance  (e.g.,  cables 
and  connectors  are  installed  to  facilitate  access); 
and  verification  that  the  software  design  for 
maintenance  and  diagnostic  routines  and  pro¬ 
cedures  are  adequate  and  the  software  can  be 
modified  to  accommodate  functional  changes. 

•  Continuity  of  Operations.  The  IOT&E  should 
provide  for  an  impact  assessment  of  the  failure 
of  any  subsystem  element  of  a  C2  system  on 
overall  mission  effectiveness. 

•  Imitative  Deception.  The  IOT&E  should 
provide  for  tests  to  assess  the  susceptibility  of 
the  data  links  of  a  C2  system  to  imitative 
deception. 

•  First  Article  Testing.  The  pre-production,  first 
article  testing  and  evaluation  should  be 
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designed  and  conducted  to:  (1)  confirm  the 
adequacy  of  the  equipment  to  meet  specified 
performance  requirements;  (2)  confirm  the  ad¬ 
equacy  of  the  software  not  only  to  meet 
current  user  needs  but  to  accommodate  chang¬ 
ing  needs;  and  (3)  determine  failure  modes  and 
rates  of  the  total  integrated  system.  This  ac¬ 
tivity  should  be  followed  by  FOT&E. 

•  Test  Planners  and  Evaluators.  Use  the  IOT&E 
personnel  in  the  FOT&E  program.  The  plan¬ 
ners  and  evaluators  for  the  FOT&E  of  the  pro¬ 
duction  system  can  do  a  better  job  if  they  are 
involved  initially  in  planning  and  conducting 
the  IOT&E. 

25.2.4  Ship  Systems 

25.2.4.1  Concept  Assessment 

•  TEMP.  Prior  to  program  initiation,  sufficient 
materiel  should  be  generated  to  allow  for  evalu¬ 
ating  the  overall  T&E  program. 

•  Test  Objectives  and  Critical  Issues.  In  evaluat¬ 
ing  the  initial  test  concept,  it  is  important  that 
the  test  objectives  during  prototype  test  and 
evaluation  address  the  major  critical  issues, 
especially  technological  issues. 

•  Test  Facilities  and  Instrumentation  Required. 
The  test  facilities  and  instrumentation  require¬ 
ments  to  conduct  developmental  and  operational 
tests  and  a  tentative  schedule  of  test  activities 
should  be  identified. 

•  Multiple  Approach  To  Weapon  System  Devel¬ 
opment.  Whenever  possible,  the  weapon  sys¬ 
tem  concept  should  not  be  predicated  on  the 
successful  development  of  a  single  hardware 
or  software  approach  in  the  various  critical  sub¬ 
systems  (unless  it  has  been  previously  demon¬ 
strated  adequately). 

•  Comparison  of  New  versus  Old  System.  The  pro¬ 
cedure  for  examining  the  relative  performance 


of  new  or  modified  systems  versus  old  should 
be  indicated  in  the  T&E  plan. 

•  Test  Support  Facilities.  The  phasing  of  test 
support  facilities  must  be  planned  carefully,  with 
some  schedule  flexibility  to  cover  late  delivery 
and  other  unforeseen  problems. 

•  Fleet  Operating  Force  Requirements.  The  re¬ 
quirement  for  fleet  operating  forces  for  DT&E 
or  OT&E  should  be  assessed  early  in  the  pro¬ 
gram  and  a  specific  commitment  made  as  to 
the  types  of  units  to  be  employed. 

•  Mission-Related  MOEs.  During  the  concept  as¬ 
sessment  of  the  acquisition  of  a  new  class  of 
ship,  a  study  effort  should  be  commenced 
jointly  by  the  Chief  of  Naval  Operations  (CNO) 
and  the  Commander,  Operational  Test  and 
Evaluation  Force  (COMOPTEVFOR).  This 
effort  is  to  establish  mission-related  measures 
of  effectiveness,  which  may  be  expressed  in 
numerical  fashion  and  may  later  be  made  the 
subject  of  OT&E  to  determine  how  closely  the 
new  ship  system  meets  the  operational  need  for 
which  it  was  conceived. 

•  Ship  T&E  Management.  The  management  of 
ship  T&E  should  ensure  that  test  requirements 
are  necessary  and  consistent  relative  to  systems/ 
subsystem  aspects  and  that  the  necessary  test¬ 
ing  is  coordinated  so  that  test  redundancy  does 
not  become  a  problem. 

•  T&E  of  Large,  Integrally-Constructed  Systems. 
Major  subsystems  should  be  proven  feasible 
before  firm  commitment  to  a  detailed  hull 
design. 

25.2.4.2  Prototype  Testing 

•  Authentication  of  Human  Factors  Concepts. 
T&E  should  authenticate  the  human  factors  con¬ 
cepts  embodied  in  the  proposed  systems  de¬ 
sign,  examining  questions  of  safety,  comfort, 
appropriateness  of  man-machine  interfaces,  as 
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well  as  the  numbers  and  skill  levels  of  the  per¬ 
sonnel  required. 

•  Acquisition  Strategy.  The  acquisition  strategy  for 
a  ship  and  its  subsystems  should  allow  sufficient 
time  between  the  planned  end  of  demonstration 
testing  and  major  procurement  decisions  of  GFE 
for  flexibility  to  modify  plans  (may  be  required 
during  the  test  phases  of  the  program). 

•  Evaluation  of  Results  of  Exploratory  Testing. 
Results  of  tests  conducted  during  exploratory 
development  and  most  likely  conducted  on 
brassboards,  breadboards,  or  modified  existing 
hardware  should  be  evaluated  carefully. 

•  Software  Testing.  In  view  of  increased  depen¬ 
dence  upon  computers  in  ship  management  and 
tactical  operation,  software  testing  must  be  ex¬ 
ceptionally  thorough,  and  integrated  software 
testing  must  begin  as  early  as  possible. 

•  New  Hull  Forms.  When  a  new  type  of  ship 
involves  a  radical  departure  from  the  conven¬ 
tional  hull  form,  extensive  prototype  testing 
should  be  required  prior  to  further  commitment 
to  the  new  hull  form. 

•  Effects  of  Hull  and  Propulsion  on  Mission  Ca¬ 
pability.  The  predicted  effects  of  the  proven  hull 
and  propulsion  system  design  on  the  perfor¬ 
mance  of  the  ship’s  critical  system  should  be 
determined. 

•  Advances  in  Propulsion.  Demonstration  of  the 
use  of  new  propulsion  systems  should  be  con¬ 
ducted  prior  to  making  the  decision  to  commit 
the  propulsion  systems  to  the  ship  in  question. 

•  Propulsion  Systems  in  Other  Classes.  When  an 
engine  to  be  used  in  the  propulsion  system  of 
a  new  ship  is  already  performing  satisfactorily 
in  another  ship,  this  is  not  to  be  taken  as  an 
indication  that  shortcuts  can  be  taken  in  pro¬ 
pulsion  system  DT&E,  or  that  no  problems  will 
be  encountered. 


•  Waivers  to  T&E  of  Ship  Systems.  Waivers  to 
T&E  of  pre-production  models  of  a  system  in 
order  to  speed  up  production  and  delivery 
should  be  made  only  after  considering  all  costs 
and  benefits  of  the  waiver,  including  those  not 
associated  with  the  contract. 

•  Environment  Effects  on  Sonar  Domes.  Envi¬ 
ronmental  effects  on  sonar  domes  and  their  self¬ 
noise  should  be  tested  and  evaluated  before  the 
domes  are  accepted  as  part  of  the  sonar  system. 

•  Hull/Machinery  Testing  by  Computer  Simulation. 
In  DT&E  ships,  there  will  be  cases  where  the 
best  means  to  conduct  evaluations  of  particular 
hull  and  machinery  capabilities  is  through  dynamic 
analysis  using  computer  simulation,  with  later 
validation  of  the  simulation  by  actual  test. 

25.2.4.3  Engineering  Development  Model 

•  Initial  or  Pilot  Phase  of  IOT&E.  Before  any 
OTs  to  demonstrate  operational  suitability  and 
effectiveness  are  conducted,  an  initial  or  pilot 
test  should  be  conducted  (Technical  Evaluation 
(TECHEVAL)). 

•  Identify  Critical  Subsystems.  In  planning  for  the 
IOT&E  of  a  ship  system,  the  critical  sub¬ 
systems,  with  respect  to  mission  performance, 
should  be  identified. 

•  Reliability  of  Critical  Systems.  T&E  should  de¬ 
termine  the  expected  reliability  at  sea  of  sys¬ 
tems  critical  to  the  ship’s  mobility  and  to  the 
primary  and  major  secondary  tasks. 

•  Consistency  in  Test  Objectives.  There  are  vari¬ 
ous  phases  in  testing  a  ship  system.  One  should 
ensure  the  objectives  of  one  phase  are  not  in¬ 
consistent  with  the  objectives  of  the  other  phases. 

•  Single  Screw  Ships.  T&E  of  the  propulsion  sys¬ 
tems  of  ships  with  a  single  screw  should  be 
especially  rigorous  to  determine  failure  rates, 
maintenance,  and  repair  alternatives. 
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•  Problems  Associated  With  New  Hulls.  When¬ 
ever  a  new  hull  is  incorporated  into  ship  design, 
a  T&E  of  this  hull  should  be  conducted  prior 
to  the  Full-Rate  Production  (FRP)  and  incor¬ 
poration  of  the  major  weapons  subsystems. 

25.2.4.4  Production  (LRIP,  Full  Rate), 
Deployment  and  Operational 
Support 

•  The  OT&E  of  Shipboard  Gun  Systems.  The  OTs 
of  shipboard  gun  systems  should  simulate  the 
stress,  exposure  time,  and  other  conditions  of 
battle  so  that  the  suitability  of  the  weapon  can 
be  evaluated  in  total. 

•  Operational  Reliability.  The  OT&E  should  pro¬ 
vide  valuable  data  on  the  operational  reliability 
of  ship  weapon  systems  that  cannot  be  ob¬ 
tained  through  DT&E. 

•  Targets  for  Antiaircraft  Warfare  (AAW)  OT&E. 
The  OT  of  shipboard  AAW  weapons  demands 
the  use  of  targets  which  realistically  simulate 
the  present-day  threat. 

•  Design  of  Ship  FOT&E.  In  the  testing  program 
of  a  ship  system,  it  should  be  recognized  that, 
although  it  may  be  designated  as  a  special- 
purpose  ship,  in  most  cases  it  will  be  used  in  a 
general-purpose  role  as  well.  This  will  cause 
post  deployment  FOT&E. 

•  Operational  Testing  During  Shakedown  Peri¬ 
ods.  The  time  period  for  FOT&E  of  a  ship  can 
be  used  more  efficiently  if  full  advantage  is 
taken  of  the  periods  immediately  after  the  ship 
is  delivered  to  the  Navy. 

•  Fleet  Operations  in  FOT&E.  A  great  deal  of 
information  on  the  operational  effectiveness  of 
a  ship  can  be  obtained  from  standard  fleet 
operations  through  well-designed  information 
collection,  processing,  and  analysis  procedures. 


•  Ship  Antisubmarine  Warfare  (ASW)  FOT&E 
Planning.  In  planning  FOT&E  of  shipboard 
systems,  it  is  important  to  recognize  the 
difficulty  of  achieving  realism,  perhaps  more 
so  than  in  other  areas  of  naval  warfare. 

•  Variable  Depth  Sonar  FOT&E.  The  behavior 
of  towed  bodies  of  variable  depth  sonar  sys¬ 
tems  and  towed  arrays  should  be  tested  and 
evaluated  under  all  ship  maneuvers  and  speeds 
likely  to  be  encountered  in  combat. 

•  Ship  Self -Noise  Tests.  The  magnetic  and  acous¬ 
tic  signatures  of  a  ship  can  be  tested  accurately 
only  after  it  is  completed. 

•  Effect  of  Major  Electronic  Countermeasures 
(ECM)  on  Ship  Capability.  The  FOT&E  of  a 
ship  should  include  tests  of  the  effectiveness 
of  the  ship  when  subjected  to  major  ECM. 

•  Ship  System  Survivability.  FOT&E  of  modem 
ships  should  provide  for  the  assessment  of  their 
ability  to  survive  and  continue  to  fight  when 
subjected  to  battle  damage. 

•  Interlocks.  Shipboard  electronic  systems  are 
designed  with  interlock  switches  that  open 
electrical  circuits  for  safety  reasons  when  the 
equipment  cabinets  are  opened.  The  FOT&E 
should  be  able  to  detect  over-design  as  well 
as  minimum  design  adequacy  of  the  interlock 
systems. 

•  Intra-ship  Communication.  In  conducting  lead 
ship  trials  and  evaluations,  particular  attention 
should  be  given  to  the  operational  impact  re¬ 
sulting  from  absence,  by  design,  of  intra-ship 
communications  circuits  and  stations  from 
important  operating  locations. 
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25.2.5  Surface  Vehicle  Systems 

25.2.5.1  Concept  Assessment 

•  Preparing  Test  Plans.  It  is  necessary  that  de¬ 
tailed  evaluation  criteria  be  established  that 
includes  all  items  to  be  tested. 

•  Test  Plans.  Prior  to  program  initiation,  a  plan 
should  be  prepared  for  evaluating  the  overall 
T&E  program.  As  part  of  this,  a  detailed  T&E 
plan  for  those  tests  to  be  conducted  before  ad¬ 
vanced  engineering  development  to  validate  the 
concept  and  hardware  approach  to  the  vehicle 
system  should  be  developed.  The  objective  of 
the  validation  test  plan  is  to  fully  evaluate  the 
performance  characteristics  of  the  new  concept 
vehicle.  This  test  plan  cannot  be  developed,  of 
course,  until  the  performance  characteristics  are 
defined. 

•  Performance  Characteristics  Range.  Stated 
performance  characteristics  derived  from  stud¬ 
ies  should  be  measured  early  in  the  program. 
Unrealistic  performance  requirements  can  lead 
to  false  starts  and  costly  delays. 

•  Operating  Degradation.  System  performance 
degrades  under  field  conditions.  Anticipated 
degradation  must  be  considered  during  T&E. 
When  a  system  must  operate  at  peak  perfor¬ 
mance  during  DT/OT  to  meet  the  specified  re¬ 
quirements,  it  will  then  be  likely  to  perform  at 
a  lesser  level  when  operated  in  the  field. 

•  Test  Personnel.  The  test  director  and/or  key 
members  of  the  test  planning  group  within  the 
project  office  should  have  significant  T&E 
experience. 

•  Design  Reviews.  T&E  factors  and  experience 
must  influence  the  system  design.  The  applica¬ 
tion  of  knowledge  derived  from  past  experi¬ 
ence  can  be  a  major  asset  in  arriving  at  a  sound 
system  design. 


•  Surrogate  Vehicles.  When  high  technical  risk 
is  present,  development  should  be  structured 
around  the  use  of  one  or  more  surrogate  ve¬ 
hicles  designed  to  prove  the  system  concept 
under  realistic  operational  conditions  before 
proceeding  with  further  development. 

•  Test  Facilities  and  Scheduling.  Test  range  and 
resource  requirements  to  conduct  validation  tests 
and  a  tentative  schedule  of  test  activities  should 
be  identified. 

25.2.5.2  Prototype  Testing 

•  Vulnerability.  The  vulnerability  of  vehicles 
should  be  estimated  on  the  basis  of  testing. 

•  Gun  and  Ammunition  Performance.  Gun  and 
ammunition  development  should  be  considered 
a  part  of  overall  tank  system  development. 
When  a  new  gun  tube,  or  one  which  has  not 
been  mounted  previously  on  a  tank  chassis,  is 
being  evaluated,  all  ammunition  types  (includ¬ 
ing  missiles)  planned  for  use  in  that  system 
should  be  test  fired  under  simulated  operational 
conditions. 

•  Increased  Complexity.  The  addition  of  new  ca¬ 
pabilities  to  an  existing  system  or  system  type 
will  generally  increase  complexity  of  the  sys¬ 
tem  and,  therefore,  increase  the  types  and 
amount  of  testing  required  and  the  time  to  per¬ 
form  these  tests. 

•  Component  Interfaces.  Prior  to  assembly  in  a 
prototype  system,  component  subsystems 
should  be  assembled  in  a  mock-up  and  veri¬ 
fied  for  physical  fit,  human  factors  consider¬ 
ations,  interface  compatibility,  and  for  electri¬ 
cal  and  mechanical  compatibility. 

•  Determining  Test  Conditions.  During  valida¬ 
tion,  test  conditions  should  be  determined  by 
the  primary  objectives  of  that  test  rather  than 
by  more  general  considerations  of  realism. 
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•  Test  Plan  Development.  The  test  plan  devel¬ 
oped  by  this  point  should  be  in  nearly  final  form 
and  include,  as  a  minimum: 

—  A  description  of  requirements, 

—  The  facilities  needed  to  make  evaluations, 
—  The  schedule  of  evaluations  and  facilities, 
—  The  reporting  procedure,  the  objective  be¬ 
ing  to  communicate  test  results  in  an  under¬ 
standable  format  to  all  program  echelons, 

—  The  T&E  guidelines,  and 
—  A  further  refinement  of  the  cost  estimates 
which  were  initiated  during  the  Concept 
Evaluation  Phase. 

•  Prototype  Tests.  Prototype  tests  should  show 
satisfactory  meeting  of  success  criteria  which 
are  meaningful  in  terms  of  operational  usage. 
It  is  essential  in  designing  contractually  required 
tests,  upon  whose  outcome  large  incentive  pay¬ 
ments  or  even  program  continuation  may  de¬ 
pend,  to  specify  broader  success  criteria  than 
simply  hit  or  miss  in  a  single  given  scenario. 

•  Reliability  Testing.  Reliability  testing  should  be 
performed  on  component  and  subsystem  as¬ 
semblies  before  testing  of  the  complete  vehicle 
system.  Prior  to  full  system  testing,  viable  com¬ 
ponent  and  subsystem  tests  should  be 
conducted. 

•  Human  Factors.  In  evaluating  ground  vehicles, 
human  factors  should  be  considered  at  all  stages 
starting  with  the  design  of  the  prototype. 

•  Test  Plan  Scheduling.  Test  plan  scheduling 
should  be  tied  to  event  milestones  rather  than 
to  the  calendar.  In  evaluating  the  adequacy  of 
the  scheduling  as  given  by  test  plans,  it  is  im¬ 
portant  that  milestones  be  tied  to  the  major 
events  of  the  weapon  system  (meeting  stated 
requirements)  and  not  the  calendar. 

•  Test  Failures.  The  T&E  schedule  should  be 
sufficiently  flexible  to  accommodate  failures 
and  correction  of  identified  problems. 


25.2.5.3  Engineering  Development  Model 

•  Pilot  and  Dry-Run  Tests.  A  scheduled  series 
of  tests  should  be  preceded  by  a  dry  run, 
which  verifies  that  the  desired  data  will  be 
obtained. 

•  Comparison  Testing.  The  test  program  should 
include  a  detailed  comparison  of  the  character¬ 
istics  of  a  new  vehicle  system  with  those  of 
existing  systems,  alternate  vehicle  system  con¬ 
cepts  (if  applicable),  and  those  of  any  system(s) 
being  replaced. 

•  Simulation.  Simulation  techniques  and  equip¬ 
ment  should  be  utilized  to  enhance  data  collec¬ 
tion.  Creation  of  histograms  for  each  test  course 
provides  a  record  of  conditions  experienced  by 
the  vehicle  during  testing.  Use  of  a  chassis  dy¬ 
namometer  can  produce  additional  driveling  en¬ 
durance  testing  with  more  complete  instrumen¬ 
tation  coverage. 

•  Environmental  Testing.  Ground  vehicles  should 
be  tested  in  environmental  conditions  and  situ¬ 
ations  comparable  to  those  in  which  they  will 
be  expected  to  perform. 

•  System  Vulnerability.  For  combat  vehicles,  some 
estimate  of  vulnerability  to  battle  damage  should 
be  made. 

•  Design  Criteria  Verification.  Subsystem  design 
criteria  should  be  compared  with  actual 
characteristics. 

•  Electromagnetic  Testing.  Vehicle  testing  should 
include  electromagnetic  testing. 

•  System  Strength  Testing.  In  evaluating  ground 
vehicles,  early  testing  should  verify  intrinsic 
strength.  This  implies  operation  with  maximum 
anticipated  loading,  including  trailed  loads  at 
maximum  speeds  and  over  worst-case  grades, 
secondary  roads  and  cross-country  conditions 
for  which  the  vehicle  was  developed  or 
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procured.  This  test  is  intended  to  identify  defi¬ 
cient  areas  of  design,  not  to  break  the 
machinery. 

•  Component  Compatibility.  Component  compat¬ 
ibility  should  be  checked  through  the  duration 
of  the  test  sequence. 

•  Human  Interface.  Critiques  of  good  and  bad 
features  of  the  vehicle  should  be  made  early  in 
the  prototype  stage  while  adequate  time  remains 
to  make  any  indicated  changes. 

•  Serviceability  Testing.  Ground  vehicles  should 
be  tested  and  evaluated  to  determine  the  rela¬ 
tive  ease  of  serviceability,  particularly  with 
high-frequency  operations. 

•  Experienced  User  Critique.  Ground  vehicle 
user  opinions  should  be  obtained  early  in  the 
development  program. 

•  Troubleshooting  During  Tests.  Provisions 
should  be  made  to  identify  subsystem  failure 
causes.  Subsystems  may  exhibit  failures  dur¬ 
ing  testing.  Adequate  provisions  should  be 


made  to  permit  troubleshooting  and  identifica¬ 
tion  of  defective  components  and  inadequate 
design. 

25.2.5.4  Production  (LRIP,  Full  Rate), 
Deployment  and  Operational 
Support 

•  Planning  the  IOT&E.  The  IOT&E  should  be 
planned  to  be  cost-effective  and  provide  mean¬ 
ingful  results. 

•  Performance  and  Reliability  Testing.  The  pro¬ 
duction  first-article  testing  should  verify  the  per¬ 
formance  of  the  vehicle  system  and  determine 
the  degradation,  failure  modes,  and  failure  rates. 

•  Lead-the-Fleet  Testing.  At  least  one  production 
prototype  or  initial  production  model  vehicle 
should  be  allocated  to  intensive  testing  to  ac¬ 
cumulate  high  operating  time  in  a  short  period. 

•  User  Evaluation.  User-reported  shortcomings 
should  be  followed  up  to  determine  problem 
areas  requiring  correction.  Fixes  should  be 
evaluated  during  a  FOT&E. 
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APPENDIX  A 

ACRONYMS  AND  THEIR  MEANINGS 


AAE 

Army  Acquisition  Executive 

AAH 

Advanced  Attack  Helicopter 

AAW 

Antiaircraft  Warfare 

ACAT 

Acquisition  Category 

ACE 

Army  Corps  of  Engineers 

ACTD 

Advanced  Concept  Technology  Demonstration 

ADM 

Acquisition  Decision  Memorandum 

ADP 

Automated  Data  Processing 

ADPE 

Automated  Data  Processing  Equipment 

AEC 

Army  Evaluation  Center 

AF 

Air  Force 

AFEWES 

Air  Force  Electronic  Warfare  Evaluation  Simulator 

AFFTC 

Air  Force  Flight  Test  Center 

AFMC 

Air  Force  Materiel  Command 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center 

AF/TE 

Chief  of  Staff,  Test  and  Evaluation  (Air  Force) 

AIS 

Automated  Information  System 

ALCM 

Air  Launch  Cruise  Missile 

AMC 

Army  Materiel  Command 

AMSDL 

Acquisition  Management  Systems  and  Data  Requirements  Control  List 

Ao 

Operational  Availability 

AoA 

Analysis  of  Alternatives 

APB 

Acquisition  Program  Baseline 

AR 

Army  Regulation 

ARL-HRED 

Army  Research  Laboratory-Human  Research  and  Engineering 
Division/Directorate 

ASA(ALT) 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and 
Technology 
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ASD(NII) 

Assistant  Secretary  of  Defense  for  Networks  and  Information 
Integration 

ASM 

Air-to-Surface  Missile 

ASN(RD&A) 

Assistant  Secretary  of  the  Navy  for  Research,  Development,  and 
Acquisition 

ASN/RE&S 

Assistant  Secretary  of  the  Navy/Research,  Engineering,  and  Science 

ASR 

Alternative  System  Review 

ASW 

Antisubmarine  Warfare 

ATD 

Advanced  Technology  Demonstration  (or  Development) 

ATE 

Automatic  Test  Equipment 

ATEC 

Army  Test  and  Evaluation  Command 

ATIRS 

Army  Test  Incident  Reporting  System 

ATPS 

Acceptance  Test  Procedures 

ATR 

Acceptance  Test  Report 

AWACS 

Airborne  Warning  and  Control  System 

BA 

Budget  Activity;  Budget  Authority 

BIT 

Built-in  Test 

BITE 

Built-in  Test  Equipment 

BLRIP 

Beyond  Low  Rate  Initial  Production  Report 

BoD 

Board  of  Directors 

BoOD 

Board  of  Operating  Directors 

C2 

Command  and  Control 

C3 

Command,  Control,  and  Communications 

C3I 

Command,  Control,  Communications,  and  Intelligence 

C4 

Command,  Control,  Communications,  and  Computers 

C4I 

Command,  Control,  Communications,  Computers,  and  Intelligence 

C4ISR 

Command,  Control,  Communications,  Intelligence,  Surveillance,  and 
Reconnaissance 

CAD 

Computer-Aided  Design 

CAE 

Component  Acquisition  Executive 

CAIV 

Cost  as  an  Independent  Variable 

CAM 

Computer-Aided  Manufacturing 
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CARS 

CDD 

CDR 

CDRL 

CE 

CEP 

CERDEC 

CFR 
CG  MCSC 
Cl 
CII 
CIO 
CJCSI 
CJCSM 
CLIN 
CNO 
CNP 
COCOM 
COI 
COIC 

COMOPTEVFOR 

COO 

CPD 

CPS 

CPU 

CR 

CSCI 

CTEIP 

CTO 

DA 


Consolidated  Acquisition  Reporting  System 
Capability  Development  Document 
Critical  Design  Review 
Contract  Data  Requirements  List 
Continuous  Education 
Circle  Error  Probability 

Communications-Electronics  Research  Development  and  Engineering 
Center 

Code  of  Federal  Regulations 

Commanding  General,  Marine  Corps  Systems  Command 
Configuration  Item 

Compatibility,  Interoperability,  and  Integration 

Chief  Information  Officer 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

Chairman  of  the  Joint  Chiefs  of  Staff  Manual 

Contract  Line  Item  Number 

Chief  of  Naval  Operations 

Candidate  Nomination  Proposal 

Combatant  Commander 

Critical  Operational  Issue 

Critical  Operational  Issues  and  Criteria 

Commander,  Operational  Test  and  Evaluation  Force  (Navy) 

Concept  of  Operations 
Capability  Production  Document 
Competitive  Prototyping  Strategy 
Central  Processing  Unit 
Concept  Refinement 
Computer  Software  Configuration  Item 
Central  Test  and  Evaluation  Investment  Program 
Comparative  Test  Office 

Developing  Agency  (Navy);  Department  of  the  Army 
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DAB 

Defense  Acquisition  Board 

DAE 

Defense  Acquisition  Executive 

DAES 

Defense  Acquisition  Executive  Summary 

DAS 

Director  of  the  Army  Staff 

DBDD 

Data  Base  Design  Document 

DCMA 

Defense  Contract  Management  Agency 

DCS 

Deputy  Chief  of  Staff 

DCS/R&D 

Deputy  Chief  of  Staff  for  Research  and  Development 

DCSOPS 

Deputy  Chief  of  Staff  for  Operations  and  Plans 

DDR&E 

Director  of  Defense  Research  and  Engineering 

DIA 

Defense  Intelligence  Agency 

DID 

Data  Item  Description 

DISA 

Defense  Information  Systems  Agency 

DLT 

Design  Limit  Test 

DMSO 

Defense  Modeling  and  Simulation  Office 

DNA 

Defense  Nuclear  Agency 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

DOE 

Department  of  Energy 

DOT&E 

Director,  Operational  Test  and  Evaluation 

DOTLPF 

Doctrine,  Organization,  Training,  Leadership,  Personnel,  and  Facilities 

DPRO 

Defense  Plant  Representative  Office 

DRR 

Defense  Readiness  Review 

DSARC 

Defense  Systems  Acquisition  Review  Council  (now  the  Defense 
Acquisition  Board  (DAB)) 

DSB 

Defense  Science  Board 

DT 

Development  Test 

DT&E 

Development  Test  and  Evaluation 

DT&E/DS 

Development  Test  and  Evaluation/Defense  Systems 

DTC 

Developmental  Test  Command  (Army) 
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DTIC 

Defense  Technical  Information  Center 

DTSE&E 

Director,  Test  Systems  Engineering  and  Evaluation 

DTTSG 

Defense  Test  and  Training  Steering  Group 

DUSA(OR) 

Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

DUSD(A&T) 

Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology 

DVAL 

Data  Link  Vulnerability  Analysis 

EA 

Evolutionary  Acquisition 

EC 

Electronic  Combat 

ECCM 

Electronic  Counter-Countermeasures 

ECM 

Electronic  Countermeasures 

ECP 

Engineering  Change  Proposal 

ECR 

Engineering  Change  Review 

EDM 

Engineering  Development  Model 

EDP 

Event  Design  Plan  (Army) 

EDT 

Engineering  Design  Test 

EFA 

Engineering  Flight  Activity 

EIS 

Environmental  Impact  Statement 

EMCOM 

Emission  Control 

EMCS 

Electromagnetic  Control  Study 

EMI 

Electromagnetic  Interference 

EMP 

Electromagnetic  Pulse 

EOA 

Early  Operational  Assessment 

ERAM 

Extended  Range  Anti-armor  Munitions 

ESM 

Electronic  Support  Measures 

ESOH 

Environmental  Safety  and  Occupational  Health 

ESS 

Environmental  Stress  Screening 

EW 

Electronic  Warfare 

FA/A 

Functional  Analysis/Allocation 

FACA 

Federal  Advisory  Committee  Act 

FAR 

Federal  Acquisition  Regulation 

FAT 

First  Article  Testing 
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FCA 

Functional  Configuration  Audit 

FCT 

Foreign  Comparative  Testing 

FDTE 

Force  Development  Tests  and  Experimentation 

FFBD 

Functional  Flow  Block  Diagram 

FOC 

Full  Operational  Capability 

FORSCOM 

Forces  Command  (Army) 

FOT&E 

Follow-on  Operational  Test  and  Evaluation 

FQR 

Formal  Qualification  Review 

FRP 

Full  Rate  Production 

FRPDR 

Full  Rate  Production  Decision  Review 

FWE 

Foreign  Weapons  Evaluation 

FY 

Fiscal  Year 

FYDP 

Future  Years  Defense  Program 

FYTP 

Future  Years  Test  Program;  Five  Year  Test  Program  (Army) 

GAO 

Government  Accounting  Office 

GFE 

Government  Furnished  Equipment 

HELSTF 

High  Energy  Laser  System  Test  Facility 

HQDA 

Headquarters,  Department  of  the  Army 

HSI 

Human  Systems  Integration 

HW 

Hardware 

HWCI 

Hardware  Configuration  Item 

HWIL 

Hardware-in-the-Loop 

ICBM 

Intercontinental  Ballistic  Missile 

ICD 

Initial  Capabilities  Document 

ICE 

Independent  Cost  Estimate 

IDD 

Interface  Decision  Document 

IEP 

Independent  Evaluation  Plan 

IER 

Independent  Evaluation  Report 

IFPP 

Information  for  Proposal  Preparation 

ILS 

Integrated  Logistics  Support 

INSCOM 

Intelligence  and  Security  Command 
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IOA 

Independent  Operational  Assessment 

IOC 

Initial  Operating  Capability 

IOT&E 

Initial  Operational  Test  and  Evaluation 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Product  Team 

IR&D 

International  Research  and  Development 

IRA 

Industrial  Resource  Analysis 

IRS 

Interface  Requirements  Specification 

IT 

Information  Technology 

ITEA 

International  Test  and  Evaluation  Association 

ITP 

Integrated  Test  Plan 

ITOP 

International  Test  Operating  Procedure 

ITSCAP 

Information  Technology  Security  Certification  and  Accreditation 
Program 

ITTS 

Instrumentation,  Targets,  and  Threat  Simulators 

IV&V 

Independent  Verification  and  Validation 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JITC 

Joint  Interoperability  Test  Command 

JLF 

Joint  Live  Fire 

JROC 

Joint  Requirements  Oversight  Council 

JT&E 

Joint  Test  and  Evaluation 

KPP 

Key  Performance  Parameter 

Kr 

Contractor 

LCC 

Life  Cycle  Cost 

LCCE 

Life  Cycle  Cost  Estimate 

LFT 

Live  Fire  Test 

LFT&E 

Live  Fire  Test  and  Evaluation 

LRIP 

Low  Rate  Initial  Production 

LSA 

Logistics  Support  Analysis 

LTBT 

Limited  Test  Ban  Treaty 

M&S 

Modeling  and  Simulation 
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MAIS 

Major  Automated  Information  System 

MAJCOM 

Major  Commands 

MANPRINT 

Manpower  and  Personnel  Integration 

MCOTEA 

Marine  Corps  Operational  Test  and  Evaluation  Activity 

MCP 

Military  Construction  Program 

MCSC 

Marine  Corps  Systems  Command 

MDA 

Milestone  Decision  Authority;  Missile  Defense  Agency 

MDAP 

Major  Defense  Acquisition  Program 

MEDCOM 

Medical  Command  (Army) 

MIL-HDBK 

Military  Handbook 

MIL-SPEC 

Military  Specification 

MIL-STD 

Military  Standard 

MMOU 

Multinational  Memorandum  of  Understanding 

MOA 

Memorandum  of  Agreement 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

MOS 

Military  Occupational  Speciality 

MOT&E 

Multi-Service  Operational  Test  and  Evaluation 

MOU 

Memorandum  of  Understanding 

MPE 

Militaiy  Preliminary  Evaluation 

MRTFB 

Major  Range  and  Test  Facility  Base 

MS 

Milestone 

MTBF 

Mean  Time  Between  Failure 

MTTR 

Mean  Time  To  Repair 

NAPMA 

North  Atlantic  Program  Management  Agency 

NATO 

North  Atlantic  Treaty  Organization 

NAVAIR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NAWC 

Naval  Air  Warfare  Center 

NBC 

Nuclear,  Biological,  and  Chemical 

NBCC 

Nuclear,  Biological,  and  Chemical  Contamination 
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NDAA 

National  Defense  Authorization  Act 

NDCP 

Navy  Decision  Coordinating  Paper 

NDI 

Nondevelopment(al)  Item 

NDU 

National  Defense  University 

NEPA 

National  Environmental  Policy  Act 

NH&S 

Nuclear  Hardness  and  Survivability 

NSIAD 

National  Security  and  International  Affairs  Division 

NSS 

National  Security  Systems 

O&M 

Operations  and  Maintenance 

O&S 

Operations  and  Support 

OA 

Operational  Assessment 

OCD 

Operational  Concept  Document 

OIPT 

Overarching  Integrated  Product  Team 

OMB 

Office  of  Management  and  Budget 

OPEVAL 

Operational  Evaluation 

OPNAV 

Operational  Navy 

OPNAVIST 

Operational  Navy  Instruction 

OPSEC 

Operations  Security 

OPTEVFOR 

Operational  Test  and  Evaluation  Force  (Navy) 

ORMAS/TE 

Operational  Resource  Management  Assessment  Systems  for  Test  and 
Evaluation 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

Operational  Test 

OT&E 

Operational  Test  and  Evaluation 

OTA 

Operational  Test  Agency 

OTC 

Operational  Test  Command  (Army) 

OTD 

Operational  Test  Director 

OTRR 

Operational  Test  Readiness  Review 

OTP 

Outline  Test  Plan 

P3I 

Preplanned  Product  Improvements 

P&D 

Production  and  Deployment 
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PAT&E 

Production  Acceptance  Test  and  Evaluation 

PCA 

Physical  Configuration  Audit 

PCO 

Primary  Contracting  Officer 

PDR 

Preliminary  Design  Review 

PDUSD(A&T) 

Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition  and 
Technology) 

PE 

Program  Element 

PEO 

Program  Executive  Officer 

PF/DOS 

Production,  Fielding/Deployment,  and  Operational  Support 

PI 

Product  Improvement 

Pk 

Probability  of  Kill 

PM 

Program  Manager;  Product  Manager 

PMO 

Program  Management  Office 

PO 

Program  Office,  Purchase  Order 

POM 

Program  Objectives  Memorandum 

PPBE 

Planning,  Programming,  Budgeting  and  Execution 

PQT 

Production  Qualification  Test 

PRAT 

Production  Reliability  Acceptance  Test 

PRESINSURV 

President  of  the  Board  of  Inspection  and  Survey 

PRR 

Production  Readiness  Review 

PSA 

Principal  Staff  Assistant 

QA 

Quality  Assurance 

QOT&E 

Qualification  Operational  Test  and  Evaluation 

R&D 

Research  and  Development 

R&E 

Research  and  Engineering 

R&M 

Reliability  and  Maintainability 

RAM 

Reliability,  Availability,  and  Maintainability 

RAS 

Requirements  Allocation  Sheet 

RCC 

Range  Commanders  Council 

RCS 

Radar  Cross  Section 

RD&A 

Research,  Development,  and  Acquisition 
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RDECOM 

Research,  Development  and  Engineering  Command  (Army) 

RDT 

Reliability  Development  Testing 

RDT&E 

Research,  Development,  Test  and  Evaluation 

RFP 

Request  for  Proposal 

RGT 

Reliability  Growth  Test 

RM 

Resource  Manager 

RQT 

Reliability  Qualification  Test 

RSI 

Rationalization,  Standardization,  and  Interoperability 

RSM 

Radar  Spectrum  Management 

S&TS 

Strategic  and  Tactical  Systems 

SAF/AQ 

Assistant  Secretary  of  the  Air  Force  for  Acquisition 

SAR 

Selected  Acquisition  Report 

SAT 

Ship  Acceptance  Test 

SDD 

Software  Design  Document;  System  Development  and  Demonstration 

SECARMY 

Secretary  of  the  Army 

SECDEF 

Secretary  of  the  Defense 

SECNAV 

Secretary  of  the  Navy 

SEP 

Systems  Engineering  Process;  System  Engineering  Plan;  System 
Evaluation  Plan  (Army) 

SFR 

System  Functional  Review 

SIL 

Software  Integration  Laboratory 

SMDC 

Space  and  Missile  Defense  Command 

SOCOM 

Special  Operations  Command 

SOO 

Statement  of  Objectives 

SOW 

Statement  of  Work 

SPAWAR 

Space  and  Naval  Warfare  Systems  Command 

SPEC 

Specification 

SPO 

System  Program  Office 

SRR 

Systems  Requirements  Review 

SRS 

Software  Requirements  Specification 

SSD 

Segment  Design  Document 
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SSR 

Software  Specification  Review 

STA 

System  Threat  Assessment 

STEP 

Simulation,  Test  and  Evaluation  Process 

STP 

Software  Test  Plan 

STRI 

Simulation,  Training,  and  Instrumentation  (Army) 

SQA 

Software  Quality  Assurance 

SVR 

System  Verification  Review 

SW 

Software 

SWIL 

Software  in  the  Loop 

T&E 

Test  and  Evaluation 

TAAF 

Test,  Analyze,  and  Fix 

TADS 

Target  Acquisition  Designation  System;  Theater  Air  Defense  System 

TAFT 

Test,  Analyze,  Fix,  and  Test 

TD 

Technology  Development 

TDS 

Technology  Development  Strategy 

TDY 

Temporary  Duty 

TEAM 

Test,  Evaluation,  Analysis,  and  Modeling 

TECHEVAL 

Technical  Evaluation  (Navy  Term) 

TEMA 

Test  and  Evaluation  Management  Agency 

TEMP 

Test  and  Evaluation  Master  Plan 

TERC 

Test  and  Evaluation  Resources  Committee 

TFRD 

Test  Facility  Requirements  Document 

TIRIC 

Training  Instrumentation  Resource  Investment  Committee 

TLS 

Time  Line  Sheet 

TM 

Technical  Manual,  Test  Manager 

TMC 

Test  Management  Council 

TPD 

Test  Program  Documentation 

TPO 

Test  Program  Outline 

TPM 

Technical  Performance  Measurement 

TPWG 

Test  Planning  Working  Group 

TRADOC 

Training  and  Doctrine  Command 
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TRIMS 

Test  Resource  Information  Management  System 

TRP 

Test  Resources  Plan 

TRR 

Test  Readiness  Review 

TRS 

Test  Requirements  Sheet 

TSARC 

Test  Schedule  and  Review  Committee 

UNK 

Unknown 

U.S. 

United  States 

USA 

United  States  Army 

USAF 

United  States  Air  Force 

USAFE/DOQ 

U.S.  Air  Force-Europe/Directorate  of  Operations-Operations 

USAKA 

United  States  Army  Kwajalein  Atoll 

USMC 

United  States  Marine  Corps 

USN 

United  States  Navy 

u.s.c. 

United  States  Code 

USD(A&T) 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 

VCJCS 

Vice  Chairman  of  the  Joint  Chiefs  of  Staff 

VCSA 

Vice  Chief  of  Staff,  Army 

VECP 

Value  Engineering  Change  Proposal 

WBS 

Work  Breakdown  Structure 

WIPT 

Working-level  Integrated  Product  Team 

WSMR 

White  Sands  Missile  Range 
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APPENDIX  B 
DOD  GLOSSARY  OF 
TEST  TERMINOLOGY 


ACCEPTANCE  TRIALS  —  Trials  and  material  inspection  conducted  underway  by  the  trail  board 
for  ships  constructed  in  a  private  shipyard  to  determine  suitability  for  acceptance  of  a  ship. 

ACQUISITION  —  The  conceptualization,  initiation,  design,  development,  test,  contracting,  produc¬ 
tion,  deployment,  Logistic  Support  (LS),  modification,  and  disposal  of  weapons  and  other  systems, 
supplies,  or  services  (including  construction)  to  satisfy  DoD  needs,  intended  for  use  in  or  in  sup¬ 
port  of  military  missions. 

ACQUISITION  CATEGORY  (ACAT)  —  ACAT  I  programs  are  Major  Defense  Acquisition  Pro¬ 
grams  (MDAPs).  An  MDAP  is  defined  as  a  program  that  is  not  highly  classified  and  is  designated 
by  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics)  (USD(AT&L))  as  an 
MDAP  or  is  estimated  by  USD(AT&L)  to  require  eventual  total  expenditure  for  research,  develop¬ 
ment,  test,  and  evaluation  of  more  than  $365  million  Fiscal  Year  (FY)2000  constant  dollars  or  for 
procurement,  of  more  than  $2,190  billion  in  FY2000  constant  dollars. 

*1.  ACAT  ID  for  which  the  Milestone  Decision  Authority  (MDA)  is  USD(AT&L).  The  “D” 
refers  to  the  Defense  Acquisition  Board  (DAB),  which  advises  the  USD(AT&L)  at  major 
decision  points. 

2.  ACAT  IC  for  which  the  MDA  is  the  DoD  Component  Head  or,  if  delegated,  the  DoD  Com¬ 
ponent  Acquisition  Executive  (CAE).  The  “C”  refers  to  Component. 

The  USD(A&T)  designates  programs  as  ACAT  ID  or  ACAT  IC. 

ACAT  IA  programs  are  Major  Automated  Information  Systems  (MAISs).  A  MAIS  is  any  program 
designated  by  the  Assistant  Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence  (ASD(C3I))  as  an  MAIS  or  is  estimated  to  require  program  costs  for  any  single  year  in 
excess  of  $32  million  in  Fiscal  Year  (FY)2000  constant  dollars,  total  program  in  excess  of  $126 
million  in  FY2000  constant  dollars,  or  total  life  cycle  costs  in  excess  of  $378  million  FY2000 
constant  dollars. 

MAIS  do  not  include  highly  sensitive  classified  programs  or  tactical  communications  systems. 

ACAT  II  program  is  a  major  system  that  is  a  combination  of  elements  that  shall  function  together  to 
produce  the  capabilities  required  to  fulfill  a  mission  need,  excluding  construction  or  other  improve¬ 
ments  to  real  property.  It  is  estimated  by  the  DoD  component  head  to  require  eventual  total  expen¬ 
diture  for  Research,  Development,  Test,  and  Evaluation  (RDT&E)  of  more  than  $140  million  in 
FY2000  constant  dollars,  or  procurement  of  more  than  $660  million  in  FY2000  constant  dollars,  or 
is  designated  as  major  by  the  DoD  Component  head. 
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ACAT  III  programs  are  defined  as  those  acquisition  programs  that  do  not  meet  the  criteria  for  an 
ACAT  I,  an  ACAT  IA,  or  an  ACAT  II.  The  MDA  is  designated  by  the  CAE  and  shall  be  at  the 
lowest  appropriate  level.  This  category  includes  less-than-major  AISs. 

ACQUISITION  DECISION  MEMORANDUM  (ADM)  —  A  memorandum  signed  by  the  Mile¬ 
stone  Decision  Authority  (MDA)  that  documents  decisions  made  as  the  result  of  a  milestone  deci¬ 
sion  review  or  in-process  review. 

ACQUISITION  LIFE  CYCLE  —  The  life  of  an  acquisition  program  consists  of  phases,  each  pre¬ 
ceded  by  a  milestone  or  other  decision  point,  during  which  a  system  goes  through  research,  devel¬ 
opment,  test  and  evaluation,  and  production.  Currently,  the  four  phases  are:  (1)  Concept  Explora¬ 
tion  (CE)  (Phase  0);  Program  Definition  and  Risk  Reduction  (PDRR)  (Phase  I);  (3)  Engineering 
and  Manufacturing  Development  (EMD)  (Phase  II);  and  (4)  Production,  Fielding/Deployment,  and 
Operational  Support  (PF/DOS)  (Phase  III). 

ACQUISITION  PHASE  —  All  the  tasks  and  activities  needed  to  bring  a  program  to  the  next  major 
milestone  occur  during  an  acquisition  phase.  Phases  provide  a  logical  means  of  progressively  trans¬ 
lating  broadly  stated  mission  needs  into  well-defined  system-specific  requirements  and  ultimately 
into  operationally  effective,  suitable,  and  survivable  systems. 

ACQUISITION  PROGRAM  BASELINE  (APB)  —  A  document  that  contains  the  most  important 
cost,  schedule,  and  performance  parameters  (both  objectives  and  thresholds)  for  the  program.  It  is 
approved  by  the  Milestone  Decision  Authority  (MDA),  and  signed  by  the  Program  Manager  (PM) 
and  his/her  direct  chain  of  supervision,  e.g.,  for  Acquisition  Categoiy  (ACAT)  ID  programs  it  is 
signed  by  the  PM,  Program  Executive  Officer  (PEO),  Component  Acquisition  Executive  (CAE), 
and  Defense  Acquisition  Executive  (DAE). 

ACQUISITION  STRATEGY  —  A  business  and  technical  management  approach  designed  to  achieve 
program  objectives  within  the  resource  constraints  imposed.  It  is  the  framework  for  planning,  di¬ 
recting,  contracting  for,  and  managing  a  program.  It  provides  a  master  schedule  for  research,  devel¬ 
opment,  test,  production,  fielding,  modification,  postproduction  management,  and  other  activities 
essential  for  program  success.  Acquisition  strategy  is  the  basis  for  formulating  functional  plans  and 
strategies  (e.g.,  Test  And  Evaluation  Master  Plan  (TEMP),  Acquisition  Plan  (AP),  competition, 
prototyping,  etc.) 

ACQUISITION  RISK  —  See  Risk. 

ADVANCED  CONCEPT  TECHNOLOGY  DEMONSTRATION  (ACTD)  —  Shall  be  used  to 
determine  military  utility  of  proven  technology  and  to  develop  the  concept  of  operations  that  will 
optimize  effectiveness.  ACTD’s  themselves  are  not  acquisition  programs,  although  they  are  de¬ 
signed  to  provide  a  residual,  usable  capability  upon  completion.  Funding  is  programmed  to  support 
2  years  in  the  field.  ACTD’s  are  funded  with  6.3a  (Advanced  Technology  Development  (or  Dem¬ 
onstration)  (ATD))  funds. 

ADVANCED  TECHNOLOGY  DEVELOPMENT  (OR  DEMONSTRATION)  (ATD)  (Budget 
Activity  6.3)  —  Projects  within  the  6.3a  (ATD)  program  which  are  used  to  demonstrate  the  matu- 
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rity  and  potential  of  advanced  technologies  for  enhanced  military  operational  capability  or  cost 
effectiveness,  It  is  intended  to  reduce  technical  risks  and  uncertainties  at  the  relatively  low  costs  of 
informal  processes. 

AGENCY  COMPONENT  —  A  major  organizational  subdivision  of  an  agency.  For  example:  the 
Army,  Navy,  Air  Force  and  Defense  Supply  Agency  are  agency  components  of  the  Department  of 
Defense.  The  Federal  Aviation,  Urban  Mass  Transportation  and  the  Federal  Highway  Administra¬ 
tions  are  agency  components  of  the  Department  of  Transportation. 

ANALYSIS  OF  ALTERNATIVES  (AoA)  —  An  analysis  of  the  estimated  costs  and  operational 
effectiveness  of  alternative  materiel  systems  to  meet  a  mission  need  and  the  associated  program  for 
acquiring  each  alternative.  Formerly  known  as  Cost  and  Operational  Effectiveness  Analysis  (COEA). 

AUTOMATED  INFORMATION  SYSTEM  (AIS)  —  A  combination  of  computer  hardware  and 
software,  data,  or  telecommunications,  that  performs  functions  such  as  collecting,  processing,  trans¬ 
mitting,  and  displaying  information.  Excluded  are  computer  resources,  both  hardware  and  soft¬ 
ware,  that  are  physically  part  of,  dedicated  to,  or  essential  in  real  time  to  the  mission  performance 
of  weapon  systems. 

AUTOMATIC  TEST  EQUIPMENT  (ATE)  —  An  equipment  that  is  designed  to  automatically  con¬ 
duct  analysis  of  functional  or  static  parameters  and  to  evaluate  the  degree  of  performance  degrada¬ 
tion  and  perform  fault  isolation  of  unit  malfunctions. 

BASELINE  —  Defined  quantity  or  quality  used  as  starting  point  for  subsequent  efforts  and  progress 
measurement  that  can  be  a  technical  cost  or  schedule  baseline. 

BRASSBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices)  used  to  de¬ 
termine  feasibility  and  to  develop  technical  and  operational  data.  It  will  normally  be  a  model  suffi¬ 
ciently  hardened  for  use  outside  of  laboratory  environments  to  demonstrate  the  technical  and  opera¬ 
tional  principles  of  immediate  interest.  It  may  resemble  the  end  item  but  is  not  intended  for  use  as 
the  end  item. 

BREADBOARD  CONFIGURATION  —  An  experimental  device  (or  group  of  devices)  used  to 
determine  feasibility  and  to  develop  technical  data.  It  will  normally  be  configured  only  for  labora¬ 
tory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  It  may  not  resemble  the  end 
item  and  is  not  intended  for  use  as  the  projected  end  item. 

CAPSTONE  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  —  A  TEMP  which  ad¬ 
dresses  the  testing  and  evaluation  of  a  program  consisting  of  a  collection  of  individual  systems 
(family  of  systems,  system  of  systems)  which  function  collectively.  Individual  system-unique  content 
requirements  are  addressed  in  an  annex  to  the  basic  Capstone  TEMP. 

CERTIFICATION  FOR  INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (IOT&E)  — 

A  service  process  undertaken  in  the  advanced  stages  of  system  development,  normally  during  low 
rate  initial  production,  resulting  in  the  announcement  of  a  system’s  readiness  to  undergo  IOT&E. 
The  process  varies  with  each  Service. 
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COMPETITIVE  PROTOTYPING  STRATEGY  (CPS)  —  Prototype  competition  between  two  or 
more  contractors  in  a  comparative  side-by-side  test. 

CONCEPT  EVALUATION  PROGRAM  (CEP)  —  The  Concept  Experimentation  Program  is  an 
annual  program  executed  by  TRADOC  for  commanders  who  have  requirements  determination 
and  force  development  missions  for  specific  warfighting  experiments.  Experiments  are  discrete 
events  investigating  either  a  material  concept  or  a  warfighting  idea.  The  CEP  procedures  are  pro¬ 
vided  in  TRADOC  Pamphlet  71-9. 

CONCURRENCY  —  Part  of  an  acquisition  strategy  which  would  combine  or  overlap  life  cycle 
phases  (such  as  Engineering  and  Manufacturing  Development  (EMD),  and  production),  or  activi¬ 
ties  (such  as  development  and  operational  testing). 

CONTINGENCY  TESTING  —  Additional  testing  required  to  support  a  decision  to  commit  added 
resources  to  a  program,  when  significant  test  objectives  have  not  been  met  during  planned  tests. 

CONTINUOUS  EVALUATION  (CE)  —  A  continuous  process,  extending  from  concept  definition 
through  deployment,  that  evaluates  the  operational  effectiveness  and  suitability  of  a  system  by 
analysis  of  all  available  data. 

COMBAT  SYSTEM  —  The  equipment,  computer  programs,  people  and  documentation  organic  to 
the  accomplishment  of  the  mission  of  an  aircraft,  surface  ship  or  submarine;  it  excludes  the  struc¬ 
ture,  material,  propulsion,  power  and  auxiliary  equipment,  transmissions  and  propulsion,  fuels  and 
control  systems,  and  silencing  inherent  in  the  construction  and  operation  of  aircraft,  surface  ships 
and  submarines. 

CONFIGURATION  MANAGEMENT  —  The  technical  and  administrative  direction  and  surveil¬ 
lance  actions  taken  to  identify  and  document  the  functional  and  physical  characteristics  of  a  Con¬ 
figuration  Item  (Cl),  to  control  changes  to  a  Cl  and  its  characteristics,  and  to  record  and  report 
change  processing  and  implementation  status.  It  provides  a  complete  audit  trail  of  decisions  and 
design  modifications. 

CONTRACT  —  An  agreement  between  two  or  more  legally  competent  parties,  in  the  proper  form, 
on  a  legal  subject  matter  or  purpose  and  for  legal  consideration. 

CONTRACTOR  LOGISTICS  SUPPORT  —  The  performance  of  maintenance  and/or  material 
management  functions  for  a  DoD  system  by  a  commercial  activity.  Historically  done  on  an  interim 
basis  until  systems  support  could  be  transitioned  to  a  DoD  organic  capability.  Current  policy  now 
allows  for  the  provision  of  system  support  by  contractors  on  a  long-term  basis.  Also  called  Long- 
Term  Contractor  Logistics  Support. 

COOPERATIVE  PROGRAMS  —  Programs  that  comprise  one  or  more  specific  cooperative  projects 
whose  arrangements  are  defined  in  a  written  agreement  between  the  parties  and  which  are  con¬ 
ducted  in  the  following  general  areas: 
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1.  Research,  Development,  Test,  and  Evaluation  (RDT&E)  of  defense  articles  (including  coop¬ 
erative  upgrade  or  other  modification  of  a  U.S.-developed  system),  joint  production  (including 
follow-on  support)  of  a  defense  article  that  was  developed  by  one  or  more  of  the  participants, 
and  procurement  by  the  United  States  of  a  foreign  defense  article  (including  software),  technol¬ 
ogy  (including  manufacturing  rights),  or  service  (including  logistics  support)  that  are  imple¬ 
mented  under  Title  22  U.S.C.  §2767,  Reference  (c),  to  promote  the  Rationalization,  Standard¬ 
ization,  and  Interoperability  (RSI)  of  North  Atlantic  Treaty  Organization  (NATO)  armed  forces 
or  to  enhance  the  ongoing  efforts  of  non-NATO  countries  to  improve  their  conventional  de¬ 
fense  capabilities. 

2.  Cooperative  Research  and  Development  (R&D)  program  with  NATO  and  major  non-NATO 
allies  implemented  under  Title  10  U.S.C.  §2350a,  to  improve  the  conventional  defense  capa¬ 
bilities  of  NATO  and  enhance  Rationalization,  Standardization,  and  Interoperability  (RSI). 

3.  Data,  information,  and  personnel  exchange  activities  conducted  under  approved  DoD  programs. 

4.  Test  and  Evaluation  (T&E)  of  conventional  defense  equipment,  munitions,  and  technologies 
developed  by  allied  and  friendly  nations  to  meet  valid  existing  U.S.  military  requirements. 

COST  AS  AN  INDEPENDENT  VARIABLE  (CATV)  —  Methodologies  used  to  acquire  and  oper¬ 
ate  affordable  DoD  systems  by  setting  aggressive,  achievable  Life  Cycle  Cost  (LCC)  objectives, 
and  managing  achievement  of  these  objectives  by  trading  off  performance  and  schedule,  as  neces¬ 
sary.  Cost  objectives  balance  mission  needs  with  projected  out-year  resources,  taking  into  account 
anticipated  process  improvements  in  both  DoD  and  industry.  CAIV  has  brought  attention  to  the 
government’s  responsibilities  for  setting/adjusting  LCC  objectives  and  for  evaluating  requirements 
in  terms  of  overall  cost  consequences. 

CRITICAL  ISSUES  —  Those  aspects  of  a  system’s  capability,  either  operational,  technical,  or  other, 
that  must  be  questioned  before  a  system’s  overall  suitability  can  be  known.  Critical  issues  are  of 
primary  importance  to  the  decision  authority  in  reaching  a  decision  to  allow  the  system  to  advance 
into  the  next  phase  of  development. 

CRITICAL  OPERATIONAL  ISSUE  (COI)  —  Operational  effectiveness  and  operational  suitabil¬ 
ity  issues  (not  parameters,  objectives,  or  thresholds)  that  must  be  examined  in  Operational  Test  and 
Evaluation  (OT&E)  to  determine  the  system’s  capability  to  perform  its  mission.  A  COI  is  normally 
phrased  as  a  question  that  must  be  answered  in  order  to  properly  evaluate  operational  effectiveness 
(e.g.,  “Will  the  system  detect  the  threat  in  a  combat  environment  at  adequate  range  to  allow  suc¬ 
cessful  engagement?”)  or  operational  suitability  (e.g.,  “Will  the  system  be  safe  to  operate  in  a 
combat  environment?”). 

DATA  SYSTEM  —  Combinations  of  personnel  efforts,  forms,  formats,  instructions,  procedures,  data 
elements  and  related  data  codes,  communications  facilities  and  automatic  data  processing  equip¬ 
ment  that  provide  an  organized  and  interconnected  means,  either  automated,  manual  or  a  mixture 
of  these  for  recording,  collecting,  processing  and  communicating  data. 
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DEFENSE  ACQUISITION  EXECUTIVE  (DAE)  —  The  individual  responsible  for  all  acquisition 
matters  within  the  DoD.  (See  DoDD  5000.1.) 

DEPARTMENT  OF  DEFENSE  ACQUISITION  SYSTEM  —  A  single  uniform  system  whereby 
all  equipment,  facilities,  and  services  are  planned,  designed,  developed,  acquired,  maintained,  and 
disposed  of  within  the  DoD.  The  system  encompasses  establishing  and  enforcing  policies  and 
practices  that  govern  acquisitions,  to  include  documenting  mission  needs  and  establishing  perfor¬ 
mance  goals  and  baselines;  determining  and  prioritizing  resource  requirements  for  acquisition  pro¬ 
grams;  planning  and  executing  acquisition  programs;  directing  and  controlling  the  acquisition  re¬ 
view  process;  developing  and  assessing  logistics  implications;  contracting;  monitoring  the  execution 
status  of  approved  programs;  and  reporting  to  the  Congress. 

DESIGNATED  ACQUISITION  PROGRAM  —  Program  designated  by  the  Director,  Operational 
Test  and  Evaluation  (DOT&E)  or  the  Deputy  Director,  Developmental  Test  and  Evaluation  (S&TS) 
for  OSD  oversight  of  test  and  evaluation. 

DEVELOPING  AGENCY  (DA)  —  The  Systems  Command  or  Chief  of  Naval  Operations  (CNO)- 
designated  project  manager  assigned  responsibility  for  the  development,  test  and  evaluation  of  a 
weapon  system,  subsystem  or  item  of  equipment. 

DEVELOPMENT  TEST  AND  EVALUATION  (DT&E)  —  T&E  conducted  throughout  the  life 
cycle  to  identify  potential  operational  and  technological  capabilities  and  limitations  of  the  alterna¬ 
tive  concepts  and  design  options  being  pursued;  support  the  identification  of  cost-performance 
tradeoffs  by  providing  analyses  of  the  capabilities  and  limitations  of  alternatives;  support  the  iden¬ 
tification  and  description  of  design  technical  risks;  assess  progress  toward  meeting  Critical  Opera¬ 
tional  Issues  (COIs),  mitigation  of  acquisition  technical  risk,  achievement  of  manufacturing  process 
requirements  and  system  maturity;  assess  validity  of  assumptions  and  conclusions  from  the  Analy¬ 
sis  of  Alternatives  (AoAs);  provide  data  and  analysis  in  support  of  the  decision  to  certify  the  system 
ready  for  Operational  Test  and  Evaluation  (OT&E);  and  in  the  case  of  Automated  Information 
Systems  (AISs),  support  an  information  systems  security  certification  prior  to  processing  classified 
or  sensitive  data  and  ensure  a  standards  conformance  certification. 

DEVELOPMENTAL  TESTER  —  The  command  or  agency  that  plans,  conducts,  and  reports  the 
results  of  developmental  testing.  Associated  contractors  may  perform  developmental  testing  on  be¬ 
half  of  the  command  or  agency. 

EARLY  OPERATIONAL  ASSESSMENT  (EOA)  —  An  operational  assessment  conducted  prior  to 
or  in  support  of  prototype  testing. 

EFFECTIVENESS  —  The  extent  to  which  the  goals  of  the  system  are  attained,  or  the  degree  to 
which  a  system  can  be  elected  to  achieve  a  set  of  specific  mission  requirements. 

ENGINEERING  CHANGE  —  An  alteration  in  the  physical  or  functional  characteristics  of  a  system 
or  item  delivered,  to  be  delivered  or  under  development,  after  establishment  of  such  characteristics. 
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ENGINEERING  CHANGE  PROPOSAL  (ECP)  —  A  proposal  to  the  responsible  authority  recom¬ 
mending  that  a  change  to  an  original  item  of  equipment  be  considered,  and  the  design  or  engineer¬ 
ing  change  be  incorporated  into  the  article  to  modify,  add  to,  delete,  or  supersede  original  parts. 

ENGINEERING  DEVELOPMENT  —  The  RDT&E  funding  category  that  includes  development 
programs  being  engineered  for  Service  use  but  not  yet  approved  for  procurement  or  operation. 
Budget  Category  6.4  includes  those  projects  in  full-scale  development  of  Service  use;  but  they 
have  not  yet  received  approval  for  production  or  had  production  funds  included  in  the  DoD  budget 
submission  for  the  budget  or  subsequent  fiscal  year. 

EVALUATION  CRITERIA  —  Standards  by  which  achievement  of  required  technical  and  opera¬ 
tional  effectiveness/suitability  characteristics  or  resolution  of  technical  or  operational  issues  may  be 
evaluated.  Evaluation  criteria  should  include  quantitative  thresholds  for  the  Initial  Operating  Capa¬ 
bility  (IOC)  system.  If  parameter  maturity  grows  beyond  IOC,  intermediate  evaluation  criteria, 
appropriately  time-lined,  must  also  be  provided. 

FIRST  ARTICLE  —  First  article  includes  pre-production  models,  initial  production  samples,  test 
samples,  first  lots,  pilot  models,  and  pilot  lots;  and  approval  involves  testing  and  evaluating  the  first 
article  for  conformance  with  specified  contract  requirements  before  or  in  the  initial  stage  of  production 
under  a  contract. 

FIRST  ARTICLE  TESTING  (FAT)  —  Production  testing  that  is  planned,  conducted,  and  moni¬ 
tored  by  the  materiel  developer.  FAT  includes  pre-production  and  initial  production  testing  con¬ 
ducted  to  ensure  that  the  contractor  can  furnish  a  product  that  meets  the  established  technical  criteria. 

FOLLOW-ON  OPERATIONAL  TEST  AND  EVALUATION  (FOT&E)  —  That  test  and  evalu 
ation  that  may  be  necessary  after  system  deployment  to  refine  the  estimates  made  during  opera¬ 
tional  test  and  evaluation,  to  evaluate  changes  and  to  re-evaluate  the  system  to  ensure  that  it  contin¬ 
ues  to  meet  operational  needs  and  retains  its  effectiveness  in  a  new  environment  or  against  a  new 
threat. 

FOLLOW-ON  PRODUCTION  TEST  — A  technical  test  conducted  subsequent  to  a  full  production 
decision  on  initial  production  and  mass  production  models  to  determine  production  conformance 
for  quality  assurance  purposes.  Program  funding  category  —  Procurement. 

FOREIGN  COMPARATIVE  TESTING  (FCT)  —  A  DoD  T&E  program  that  is  prescribed  in 
Title  10  U.S.C.  §2350a(g),  and  is  centrally  managed  by  the  Director,  Foreign  Comparative  Test 
(Strategic  and  Tactical  Systems  (S&TS)).  It  provides  funding  for  U.S.  T&E  of  selected  equipment 
items  and  technologies  developed  by  allied  countries  when  such  items  and  technologies  are  identi¬ 
fied  as  having  good  potential  to  satisfy  valid  DoD  requirements. 

FUTURE- YEAR  DEFENSE  PROGRAM  (FYDP)  —(Formerly  the  Five  Year  Defense  Program). 
The  official  DoD  document  which  summarizes  forces  and  resources  associated  with  programs 
approved  by  the  Secretary  of  Defense  (SECDEF).  Its  three  parts  are  the  organizations  affected, 
appropriations  accounts  (Research,  Development,  Test,  and  Evaluation  (RDT&E),  Operations  and 
Maintenance  (O&M),  etc.),  and  the  1 1  major  force  programs  (strategic  forces,  airlift/sealift,  Research 
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and  Development  (R&D),  etc.).  R&D  is  Program  06.  Under  the  current  Planning,  Programming, 
and  Budgeting  and  Execution  (PPBE)  cycle,  the  FYDP  is  updated  when  the  services  submit  their 
Program  Objective  Memorandums  (POMs)  to  the  Office  of  the  Secretary  of  Defense  (OSD)  (May/ 
June),  when  the  services  submit  their  budgets  to  OSD  (September),  and  when  the  President  sub¬ 
mits  the  national  budget  to  the  Congress  (February).  The  primary  data  element  in  the  FYDP  is  the 
Program  Element  (PE). 

HARMONIZATION  —  Refers  to  the  process,  or  results,  of  adjusting  differences  or  inconsistencies 
in  the  qualitative  basic  military  requirements  of  the  United  States,  its  allies,  and  other  friendly 
countries.  It  implies  that  significant  features  will  be  brought  into  line  so  as  to  make  possible  sub¬ 
stantial  gains  in  terms  of  the  overall  objectives  of  cooperation  (e.g.,  enhanced  utilization  of  re¬ 
sources,  standardization,  and  compatibility  of  equipment).  It  implies  especially  that  comparatively 
minor  differences  in  “requirements”  should  not  be  permitted  to  serve  as  a  basis  for  the  support  of 
slightly  different  duplicative  programs  and  projects. 

HUMAN  SYSTEMS  INTEGRATION  —  A  disciplined,  unified,  and  interactive  approach  to  inte¬ 
grate  human  considerations  into  system  design  to  improve  total  system  performance  and  reduce 
costs  of  ownership.  The  major  categories  of  human  considerations  are  manpower,  personnel,  train¬ 
ing,  human  factors  engineering,  safety,  and  health. 

INDEPENDENT  EVALUATION  REPORT  —  A  report  that  provides  an  assessment  of  item  or 
system  operational  effectiveness  and  operational  suitability  versus  critical  issues  as  well  as  the  ad¬ 
equacy  of  testing  to  that  point  in  the  development  of  item  or  system. 

INDEPENDENT  OPERATIONAL  TEST  AGENCY  —  The  Army  Operational  Test  Command 
(OTC),  the  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR),  the  Air  Force  Operational 
Test  and  Evaluation  Center  (AFOTEC),  the  Marine  Corps  Operational  Test  and  Evaluation  Activ¬ 
ity  (MCOTEA),  and  for  the  Defense  Information  Systems  Agency  (DISA)  -  the  Joint  Interoper¬ 
ability  Test  Command  (JITC). 

INDEPENDENT  VERIFICATION  AND  VALIDATION  (TV&V)  —  An  independent  review  of 
the  software  product  for  functional  effectiveness  and  technical  sufficiency. 

INITIAL  OPERATIONAL  CAPABILITY  (IOC)  —  The  first  attainment  of  the  capability  to  employ 
effectively  a  weapon,  item  of  equipment,  or  system  of  approved  specific  characteristics  with  the 
appropriate  number,  type,  and  mix  of  trained  and  equipped  personnel  necessary  to  operate,  maintain, 
and  support  the  system.  It  is  normally  defined  in  the  Operational  Requirements  Document  (ORD). 

INITIAL  OPERATIONAL  TEST  AND  EVALUATION  (IOT&E)  —  Operational  test  and  evalu¬ 
ation  conducted  on  production,  or  production  representative  articles,  to  determine  whether  systems 
are  operationally  effective  and  suitable  for  intended  use  by  representative  users  to  support  the  deci¬ 
sion  to  proceed  beyond  Low  Rate  Initial  Production  (LRIP).  For  the  Navy,  see  Operational 
Evaluation  (OPEVAL). 

IN-PROCESS  REVIEW  —  Review  of  a  project  or  program  at  critical  points  to  evaluate  status  and 
make  recommendations  to  the  decision  authority. 
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INSPECTION  — Visual  examination  of  the  item  (hardware  and  software)  and  associated  descriptive 
documentation  which  compares  appropriate  characteristics  with  predetermined  standards  to  deter¬ 
mine  conformance  to  requirements  without  the  use  of  special  laboratory  equipment  or  procedures. 

INTEGRATED  PRODUCT  AND  PROCESS  DEVELOPMENT  (IPPD)  —  A  management  tech¬ 
nique  that  simultaneously  integrates  all  essential  acquisition  activities  through  the  use  of 
multidisciplinary  teams  to  optimize  the  design,  manufacturing,  and  supportability  processes.  IPPD 
facilitates  meeting  cost  and  performance  objectives  from  product  concept  through  production,  in¬ 
cluding  field  support.  One  of  the  key  IPPD  tenets  is  multidisciplinary  teamwork  through  Integrated 
Product  Teams  (IPTs). 

INTEGRATED  PRODUCT  TEAM  (IPT)  —  Team  composed  of  representatives  from  all  appropri¬ 
ate  functional  disciplines  working  together  to  build  successful  programs,  identify  and  resolve  is¬ 
sues,  and  make  sound  and  timely  recommendations  to  facilitate  decision  making.  There  are  three 
types  of  IPTs:  Overarching  IPTs  (OIPTs)  focus  on  strategic  guidance,  program  assessment,  and 
issue  resolution;  Working-level  IPTs  (WIPTs)  identify  and  resolve  program  issues,  determine  pro¬ 
gram  status,  and  seek  opportunities  for  acquisition  reform;  and  program  level  IPTs  focus  on  pro¬ 
gram  execution  and  may  include  representatives  from  both  government  and  after  contract  award 
industry. 

INTEROPERABILITY  —  The  ability  of  systems,  units,  or  forces  to  provide  services  to  or  accept 
services  from  other  systems,  units,  or  forces  and  to  use  the  services  so  exchanged  to  operate  effec¬ 
tively  together.  The  conditions  achieved  among  communications-electronics  systems  or  items  of 
communications-electronics  equipment  when  information  or  services  can  be  exchanged  directly 
and  satisfactorily  between  them  and/or  their  users.  Designated  an  element  of  the  Net-Ready  Key 
Performance  Parameter  (KPP)  by  Joint  Chiefs  of  Staff  (JCS). 

ISSUES  — Any  aspect  of  the  system’s  capability,  either  operational,  technical  or  other,  that  must  be 
questioned  before  the  system’s  overall  military  utility  can  be  known.  Operational  issues  are  issues 
that  must  be  evaluated  considering  the  soldier  and  the  machine  as  an  entity  to  estimate  the  opera¬ 
tional  effectiveness  and  operational  suitability  of  the  system  in  its  complete  user  environment. 

KEY  PERFORMANCE  PARAMETERS  (KPPs)  —  Those  minimum  attributes  or  characteristics 
considered  most  essential  for  an  effective  military  capability.  The  Capability  Development  Docu¬ 
ment  (CDD)  and  the  Capability  Production  Document  (CPD)  KPPs  are  included  verbatim  in  the 
Acquisition  Program  Baseline  (APB). 

LIVE  FIRE  TEST  AND  EVALUATION  (LFT&E)  —  A  test  process  that  is  defined  in  Title  10 
U.S.C.  §2366,  that  must  be  conducted  on  a  covered  system,  major  munition  program,  missile 
program,  or  Product  Improvement  (PI)  to  a  covered  system,  major  munition  program,  or  missile 
program  before  it  can  proceed  beyond  Low  Rate  Initial  Production  (LRIP).  A  covered  system  is  a 
major  system  (Acquisition  Category  (ACAT)  I,  II)  that  is;  1)  user  occupied  and  designed  to  provide 
some  degree  of  protection  to  its  occupants  in  combat,  or ;  2)  a  conventional  munitions  program  or 
missile  program,  or;  3)  a  conventional  munitions  program  for  which  more  than  1,000,000,000 
rounds  are  planned  to  be  acquired,  or;  4)  a  modification  to  a  covered  system  that  is  likely  to  affect 
significantly  the  survivability  or  lethality  of  such  a  system. 
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LIVE  FIRE  TEST  AND  EVALUATION  (LFT&E)  REPORT  —  Report  prepared  by  the  Director, 
Operational  Test  and  Evaluation  (DOT&E)  on  survivability  and  lethality  testing.  Submitted  to  the 
Congress  for  covered  systems  prior  to  the  decision  to  proceed  beyond  Low  Rate  Initial  Production 
(LRIP). 

LETHALITY  —  The  probability  that  weapon  effects  will  destroy  the  target  or  render  it  neutral. 

LIFE-CYCLE  COST  (LCC)  —  The  total  cost  to  the  government  for  the  development,  acquisition, 
operation,  and  logistic  support  of  a  system  or  set  of  forces  over  a  defined  life  span. 

LOGISTICS  SUPPORTAMLITY  —  The  degree  of  ease  to  which  system  design  characteristics  and 
planned  logistics  resources  (including  the  Logistics  Support  (LS)  elements)  allow  for  the  meeting 
of  system  availability  and  wartime  usage  requirements. 

LONG  LEAD  ITEMS  —  Those  components  of  a  system  or  piece  of  equipment  that  take  the  longest 
time  to  procure  and,  therefore,  may  require  an  early  commitment  of  funds  in  order  to  meet  acquisi¬ 
tion  program  schedules. 

LOT  ACCEPTANCE  —  This  test  is  based  on  a  sampling  procedure  to  ensure  that  the  product  retains 
its  quality.  No  acceptance  or  installation  should  be  permitted  until  this  test  for  the  lot  has  been 
successfully  completed. 

LOW  RATE  INITIAL  PRODUCTION  (LRIP)  —  The  minimum  number  of  systems  (other  than 
ships  and  satellites)  to  provide  production  representative  articles  for  Operational  Test  and  Evalua¬ 
tion  (OT&E),  to  establish  an  initial  production  base,  and  to  permit  an  orderly  increase  in  the  pro¬ 
duction  rate  sufficient  to  lead  to  Full-Rate  Production  (FRP)  upon  successful  completion  of  opera¬ 
tional  testing.  For  Major  Defense  Acquisition  Programs  (MDAPs),  LRIP  quantities  in  excess  of  10 
percent  of  the  acquisition  objective  must  be  reported  in  the  Selected  Acquisition  Report  (SAR).  For 
ships  and  satellites  LRIP  is  the  minimum  quantity  and  rate  that  preserves  mobilization. 

MAINTAINABILITY  —  The  ability  of  an  item  to  be  retained  in,  or  restored  to,  a  specified  condition 
when  maintenance  is  performed  by  personnel  having  specified  skill  levels,  using  prescribed  procedures 
and  resources,  at  each  prescribed  level  of  maintenance  and  repair.  (See  Mean  Time  To  Repair  (MTTR)) 

MAJOR  DEFENSE  ACQUISITION  PROGRAM  —  See  “Acquisition  Category.” 

MAJOR  SYSTEM  (DoD)  —  See  “Acquisition  Category.” 

MAJOR  RANGE  AND  TEST  FACILITY  BASE  (MRTFB)  —  The  complex  of  major  DoD  ranges 
and  test  facilities  managed  according  to  DoD  3200.1 1  by  the  Director,  Test  Resource  Management 
Center  reporting  to  the  USD(AT&L). 

MEAN  TIME  BETWEEN  FAILURE  (MTBF)  —  For  a  particular  interval,  the  total  functional  life 
of  a  population  of  an  item  divided  by  the  total  number  of  failures  within  the  population.  The 
definition  holds  for  time,  rounds,  miles,  events,  or  other  measures  of  life  unit.  A  basic  technical 
measure  of  reliability. 
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MEAN  TIME  TO  REPAIR  (MTTR)  —  The  total  elapsed  time  (clock  hours)  for  corrective  mainte¬ 
nance  divided  by  the  total  number  of  corrective  maintenance  actions  during  a  given  period  of  time. 
A  basic  technical  measure  of  maintainability. 

MEASURE  OF  EFFECTIVENESS  (MOE) — A  measure  of  operational  performance  success  that  must 
be  closely  related  to  the  objective  of  the  mission  or  operation  being  evaluated.  For  example,  kills  per 
shot,  probability  of  kill,  effective  range,  etc.  Linkage  shall  exist  among  the  various  MOEs  used  in  the 
Analysis  of  Alternatives  (AoAs),  Operations  Requirements  Document  (ORD)  and  Test  and  Evaluation 
(T&E);  in  particular,  the  MOEs,  Measures  of  Performance  (MOPs),  criteria  in  the  ORD,  the  AoAs,  the 
Test  and  Evaluation  Master  Plan  (TEMP)  and  the  Acquisition  Program  Baseline  (APB)  shall  be  consis¬ 
tent.  A  meaningful  MOE  must  be  quantifiable  and  a  measure  of  what  degree  the  real  objective  is  achieved. 

MEASURE  OF  PERFORMANCE  (MOP)  —  Measure  of  a  lower  level  of  performance  represent¬ 
ing  subsets  of  Measure  of  Effectiveness  (MOEs).  Examples  are  speed,  payload,  range,  time  on 
station,  frequency,  or  other  distinctly  quantifiable  performance  features. 

MILESTONE  —  The  point  when  a  recommendation  is  made  and  approval  sought  regarding  starting 
or  continuing  (proceeding  to  next  phase)  an  acquisition  program. 

MILESTONE  DECISION  AUTHORITY  (MDA)  —  The  individual  designated  in  accordance  with 
criteria  established  by  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics) 
(USD(AT&L)),  or  by  the  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence)  (ASD(C3I))  for  Automated  Information  System  (AIS)  acquisition  programs  (DoD  5000.2- 
R  (Reference  C)),  to  approve  entry  of  an  acquisition  program  into  the  next  acquisition  phase. 

MILITARY  OPERATIONAL  REQUIREMENT  —  The  formal  expression  of  a  military  need, 
responses  to  which  results  in  development  or  acquisition  of  item,  equipment’s,  or  systems. 

MISSION  RELIABILITY  —  The  probability  that  a  system  will  perform  mission  essential  functions 
for  a  given  period  of  time  under  conditions  stated  in  the  mission  profile. 

MODEL  — A  model  is  a  representation  of  an  actual  or  conceptual  system  that  involves  mathematics, 
logical  expressions,  or  computer  simulations  that  can  be  used  to  predict  how  the  system  might 
perform  or  survive  under  various  conditions  or  in  a  range  of  hostile  environments. 

MULTI-SERVICE  TEST  AND  EVALUATION  —  T&E  conducted  by  two  or  more  DoD  compo¬ 
nents  for  systems  to  be  acquired  by  more  than  one  DoD  component  (Joint  acquisition  program),  or 
for  a  DoD  component’s  systems  that  have  interfaces  with  equipment  of  another  DoD  component. 
May  be  developmental  testing  or  operational  testing  (MOT&E). 

NONDEVELOPMENT  ITEM  (NDI)  —  A  nondevelopmental  item  is  any  previously  developed 
item  of  supply  used  exclusively  for  government  purposes  by  a  federal  agency,  a  state  or  local 
government,  or  a  foreign  government  with  which  the  United  States  has  a  mutual  defense  coopera¬ 
tion  agreement;  any  item  described  above  that  requires  only  minor  modifications  or  modifications 
of  the  type  customarily  available  in  the  commercial  marketplace  in  order  to  meet  the  requirements 
of  the  processing  department  or  agency. 
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NONMAJOR  DEFENSE  ACQUISITION  PROGRAM  —A  program  other  than  a  Major  Defense 
Acquisition  Program  (MDAP)  Acquisition  Category  (ACAT)  I  or  a  highly  sensitive  classified  pro¬ 
gram:  i.e.,  ACAT  II  and  ACAT  III  programs. 

NUCLEAR  HARDNESS  — A  quantitative  description  of  the  resistance  of  a  system  or  component  to 
malfunction  (temporary  and  permanent)  and/or  degraded  performance  induced  by  a  nuclear  weapon 
environment.  Measured  by  resistance  to  physical  quantities  such  as  overpressure,  peak  velocities, 
energy  absorbed,  and  electrical  stress.  Hardness  is  achieved  through  adhering  to  appropriate  design 
specifications  and  is  verified  by  one  or  more  test  and  analysis  techniques. 

OBJECTIVE  —  The  performance  value  that  is  desired  by  the  user  and  which  the  Program  Manager 
(PM)  is  attempting  to  obtain.  The  objective  value  represents  an  operationally  meaningful,  time 
critical,  and  cost  effective  increment  above  the  performance  threshold  for  each  program  parameter. 

OPEN  SYSTEMS  — Acquisition  of  Weapons  Systems  An  integrated  technical  and  business  strategy 
that  defines  key  interfaces  for  a  system  (or  a  piece  of  equipment  under  development)  in  accordance 
with  those  adopted  by  formal  consensus  bodies  (recognized  industry  standards’  bodies)  as  specifi¬ 
cations  and  standards,  or  commonly  accepted  (de  facto)  standards  (both  company  proprietary  and 
non-proprietary)  if  they  facilitate  utilization  of  multiple  suppliers. 

OPERATIONAL  ASSESSMENT  (OA)  —  An  evaluation  of  operational  effectiveness  and  opera¬ 
tional  suitability  made  by  an  independent  operational  test  activity,  with  user  support  as  required,  on 
other  than  production  systems.  The  focus  of  an  OA  is  on  significant  trends  noted  in  development 
efforts,  programmatic  voids,  areas  of  risk,  adequacy  of  requirements,  and  the  ability  of  the  program 
to  support  adequate  Operational  Testing  (OT).  OA  may  be  made  at  any  time  using  technology 
demonstrators,  prototypes,  mock-ups,  engineering  development  models,  or  simulations  but  will  not 
substitute  for  the  independent  Operational  Test  and  Evaluation  (OT&E)  necessary  to  support  full 
production  decisions. 

OPERATIONAL  AVAILABILITY  (Ao)  —  The  degree  (expressed  in  terms  of  1.0  or  100  percent  as 
the  highest)  to  which  one  can  expect  an  equipment  or  weapon  systems  to  work  properly  when  it  is 
required.  The  equation  is  uptime  over  uptime  plus  downtime,  expressed  as  Ao.  It  is  the  quantitative 
link  between  readiness  objectives  and  supportability. 

OPERATIONAL  EFFECTIVENESS  —  The  overall  degree  of  mission  accomplishment  of  a  sys¬ 
tem  when  used  by  representative  personnel  in  the  environment  planned  or  expected  (e.g.  natural, 
electronic,  threat,  etc.)  for  operational  employment  of  the  system  considering  organization,  doc¬ 
trine,  tactics,  survivability,  vulnerability  and  threat  (including  countermeasures;  initial  nuclear  weapons 
effects;  and  Nuclear,  Biological,  and  Chemical  Contamination  (NBCC)  threats). 

OPERATIONAL  EVALUATION  —  Addresses  the  effectiveness  and  suitability  of  the  weapons, 
equipment  or  munitions  for  use  in  combat  by  typical  military  users  and  the  system  operational 
issues  and  criteria;  provides  information  to  estimate  organizational  structure,  personnel  require¬ 
ments,  doctrine,  training  and  tactics;  identifies  any  operational  deficiencies  and  the  need  for  any 
modifications;  and  assesses  MANPRINT  (safety,  health  hazards,  human  factors,  manpower,  and 
personnel)  aspects  of  the  system  in  a  realistic  operational  environment. 
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OPERATIONAL  REQUIREMENTS  —  User-or  user  representative-generated  validated  needs  de¬ 
veloped  to  address  mission  area  deficiencies,  evolving  threats,  emerging  technologies  or  weapon 
system  cost  improvements.  Operational  requirements  form  the  foundation  for  weapon  system  unique 
specifications  and  contract  requirements. 

OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD)  —  Documents  the  users  objectives 
and  minimum  acceptable  requirements  for  operational  performance  of  a  proposed  concept  or  system. 
Format  is  contained  in  the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01B, 
Requirements  Generation  System  (RGS),  15  April  2001. 

OPERATIONAL  SUITABILITY  —  The  degree  to  which  a  system  can  be  placed  satisfactorily  in 
field  use  with  consideration  being  given  to  availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human  factors,  manpower  supportability,  logis¬ 
tic  supportability,  natural  environmental  effects  and  impacts,  documentation,  and  training  requirements. 

OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  —  The  field  test,  under  realistic  condi¬ 
tions,  of  any  item  (or  key  component)  of  weapons,  equipment,  or  munitions  for  the  purpose  of 
determining  the  effectiveness  and  suitability  of  the  weapons,  equipment,  or  munitions  for  use  in 
combat  by  typical  military  users;  and  the  evaluation  of  the  results  of  such  tests.  (10  U.S.C.  2399) 

OPERATIONAL  TEST  CRITERIA  —  Expressions  of  the  operational  level  of  performance  re¬ 
quired  of  the  military  system  to  demonstrate  operational  effectiveness  for  given  functions  during 
each  operational  test.  The  expression  consists  of  the  function  addressed,  the  basis  for  comparison, 
the  performance  required  and  the  confidence  level. 

OPERATIONAL  TEST  READINESS  REVIEW  (OTRR)  —  A  review  to  identify  problems  that 
may  impact  the  conduct  of  an  Operational  Test  and  Evaluation  (OT&E).  The  OTRRs  are  con¬ 
ducted  to  determine  changes  required  in  planning,  resources  or  testing  necessary  to  proceed  with 
the  OT&E.  Participants  include  the  operational  tester  (chair),  evaluator,  material  developer,  user 
representative,  logisticians,  Headquarters,  Department  of  the  Army  (HQDA)  staff  elements  and 
others  as  necessary. 

PARAMETER  — A  determining  factor  or  characteristic.  Usually  related  to  performance  in  developing 
a  system. 

PERFORMANCE  —  Those  operational  and  support  characteristics  of  the  system  that  allow  it  to 
effectively  and  efficiently  perform  its  assigned  mission  over  time.  The  support  characteristics  of  the 
system  include  both  supportability  aspects  of  the  design  and  the  support  elements  necessary  for 
system  operation. 

PILOT  PRODUCTION  —  Production  line  normally  established  during  first  production  phase  to  test 
new  manufacturing  methods  and  procedures.  Normally  funded  by  Research,  Development,  Test, 
and  Evaluation  (RDT&E)  until  the  line  is  proven. 

POSTPRODUCTION  TESTING  —  Testing  conducted  to  assure  that  materiel  that  is  reworked, 
repaired,  renovated,  rebuilt  or  overhauled  after  initial  issue  and  deployment  conforms  to  specified 
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quality,  reliability,  safety  and  operational  performance  standards.  Included  in  postproduction  tests 
are  surveillance  tests,  stockpile  reliability  and  reconditioning  tests. 

PREPLANNED  PRODUCT  IMPROVEMENT  (1*1)  —  Planned  future  evolutionary  improvement 
of  developmental  systems  for  which  design  considerations  are  effected  during  development  to  en¬ 
hance  future  application  of  projected  technology.  Includes  improvements  planned  for  ongoing  sys¬ 
tems  that  go  beyond  the  current  performance  envelope  to  achieve  a  needed  operational  capability. 

PREPRODUCTION  PROTOTYPE  —  An  article  in  final  form  employing  standard  parts,  represen¬ 
tative  of  articles  to  be  produced  subsequently  in  a  production  line. 

PREPRODUCTION  QUALIFICATION  TEST  —  The  formal  contractual  tests  that  ensure  design 
integrity  over  the  specified  operational  and  environmental  range.  These  tests  usually  use  prototype 
or  pre-production  hardware  fabricated  to  the  proposed  production  design  specifications  and  draw¬ 
ings.  Such  tests  include  contractual  Reliability  and  Maintainability  (R&M)  demonstrations  tests 
required  prior  to  production  release. 

PROBABILITY  OF  KILL  (Pk)  —  The  lethality  of  a  weapon  system.  Generally  refers  to  arma¬ 
ments.  (e.g.,  missiles,  ordnance,  etc.).  Usually  the  statistical  probability  that  the  weapon  will  deto¬ 
nate  close  enough  to  the  target  with  enough  effectiveness  to  disable  the  target. 

PRODUCT  IMPROVEMENT  (PI)  —  Effort  to  incorporate  a  configuration  change  involving  engi¬ 
neering  and  testing  effort  on  end  items  and  depot  repairable  components,  or  changes  on  other  than 
developmental  items  to  increase  system  or  combat  effectiveness  or  extend  useful  military  fife.  Usually 
results  from  feedback  from  the  users. 

PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION  (PAT&E)  —  T&E  of  production 
items  to  demonstrate  that  items  procured  fulfill  requirements  and  specifications  of  the  procuring 
contract  or  agreements. 

PRODUCTION  ARTICLE  —  The  end  item  under  initial  or  full  rate  production. 

PRODUCTION  PROVE  OUT  — A  technical  test  conducted  prior  to  production  testing  with  proto¬ 
type  hardware  to  determine  the  most  appropriate  design  alternative.  This  testing  may  also  provide 
data  on  safety,  the  achievability  of  critical  system  technical  characteristics,  refinement  and  rugge- 
dization  of  hardware  configurations,  and  determination  of  technical  risks. 

PRODUCTION  QUALIFICATION  TEST  (PQT)  —  A  technical  test  completed  prior  to  the  full 
rate  production  decision  to  ensure  the  effectiveness  of  the  manufacturing  process,  equipment,  and 
procedures.  This  testing  also  serves  the  purpose  of  providing  data  for  the  independent  evaluation 
required  for  materiel  release  so  that  the  evaluator  can  address  the  adequacy  of  the  materiel  with 
respect  to  the  stated  requirements.  These  test  are  conducted  on  a  number  of  samples  taken  at  ran¬ 
dom  from  the  first  production  lot,  and  are  repeated  if  the  process  or  design  is  changed  significantly, 
and  when  a  second  or  alternative  source  is  brought  on  line. 
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PROGRAM  MANAGER  (PM)  —  A  military  or  civilian  official  who  is  responsible  for  managing, 
through  Integrated  Product  Teams  (IPTs),  an  acquisition  program. 

PROTOTYPE  — An  original  or  model  on  which  a  later  system/item  is  formed  or  based.  Early  proto¬ 
types  may  be  built  during  early  design  stages  and  tested  prior  to  advancing  to  advanced  engineer¬ 
ing.  Selected  prototyping  may  evolve  into  an  Engineering  Development  Model  (EDM),  as  required 
to  identify  and  resolve  specific  design  and  manufacturing  risks  in  that  phase  or  in  support  of  Pre¬ 
planned  Product  Improvement  (P3I)  or  Evolutionary  Acquisition  (EA). 

QUALIFICATIONS  TESTING  —  Simulates  defined  operational  environmental  conditions  with  a 
predetermined  safety  factor,  the  results  indicating  whether  a  given  design  can  perform  its  function 
within  the  simulated  operational  environment  of  a  system. 

QUALITY  ASSURANCE  (QA)  —  A  planned  and  systematic  pattern  of  all  actions  necessary  to 
provide  confidence  that  adequate  technical  requirements  are  established,  that  products  and  services 
conform  to  established  technical  requirements,  and  that  satisfactory  performance  is  achieved. 

REALISTIC  TEST  ENVIRONMENT  —  The  conditions  under  which  the  system  is  expected  to  be 
operated  and  maintained,  including  the  natural  weather  and  climatic  conditions,  terrain  effects, 
battlefield  disturbances,  and  enemy  threat  conditions. 

RELIABILITY  —  The  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure,  degra¬ 
dation,  or  demand  on  the  support  system.  (See  Mean  Time  Between  Failure  (MTBF).) 

REQUIRED  OPERATIONAL  CHARACTERISTICS  —  System  parameters  that  are  primary  indi¬ 
cators  of  the  system’s  capability  to  be  employed  to  perform  the  required  mission  functions,  and  to  be 
supported. 

REQUIRED  TECHNICAL  CHARACTERISTICS  —  System  parameters  selected  as  primary  in¬ 
dicators  of  achievement  of  engineering  goals.  These  need  not  be  direct  measures  of,  but  should 
always  relate  to  the  system’s  capability  to  perform  the  required  mission  functions,  and  to  be  supported. 

RESEARCH  —  1.  Systematic  inquiry  into  a  subject  in  order  to  discover  or  revise  facts,  theories,  etc. 
to  investigate.  2.  Means  of  developing  new  technology  for  potential  use  in  defense  systems. 

RISK  —  A  measure  of  the  inability  to  achieve  program  objectives  within  defined  cost  and  schedule 
constraints.  Risk  is  associated  with  all  aspects  of  the  program,  e.g.,  threat,  technology,  design  pro¬ 
cesses,  Work  Breakdown  Structure  (WBS)  elements,  etc.  It  has  two  components: 

1 .  The  probability  of  failing  to  achieve  a  particular  outcome;  and 

2.  The  consequences  of  failing  to  achieve  that  outcome. 

RISK  ASSESSMENT  —  The  process  of  identifying  program  risks  within  risk  areas  and  critical 
technical  processes,  analyzing  them  for  their  consequences  and  probabilities  of  occurrence,  and 
prioritizing  them  for  handling. 
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RISK  MONITORING  —  A  process  that  systematically  tracks  and  evaluates  the  performance  of  risk 
items  against  established  metrics  throughout  the  acquisition  process  and  develops  further  risk  reduction 
handling  options  as  appropriate. 

SAFETY  —  The  application  of  engineering  and  management  principles,  criteria,  and  techniques  to 
optimize  safety  within  the  constraints  of  operational  effectiveness,  time,  and  cost  throughout  all 
phases  of  the  system  life  cycle. 

SAFETY/HEALTH  VERIFICATION  —  The  development  of  data  used  to  evaluate  the  safety  and 
health  features  of  a  system  to  determine  its  acceptability.  This  is  done  primarily  during  Develop¬ 
mental  Test  (DT)  and  user  or  Operational  Test  (OT)  and  evaluation  and  supplemented  by  analysis 
and  independent  evaluations. 

SAFETY  RELEASE  —  A  formal  document  issued  to  a  user  test  organization  before  any  hands-on 
use  or  maintenance  by  personnel.  The  safety  release  indicates  the  system  is  safe  for  use  and  main¬ 
tenance  by  typical  user  personnel  and  describes  the  system  safety  analyses.  Operational  limits  and 
precautions  are  included.  The  test  agency  uses  the  data  to  integrate  safety  into  test  controls  and 
procedures  and  to  determine  if  the  test  objectives  can  be  met  within  these  limits. 

SELECTED  ACQUISITION  REPORT  —  Standard,  comprehensive,  summary  status  reports  on 
major  defense  acquisition  programs  (MDAPs)  (Acquisition  Category  (ACAT)  I)  required  for  peri¬ 
odic  submission  to  the  Congress.  They  include  key  cost,  schedule,  and  technical  information. 

SIMULATION  —  A  simulation  is  a  method  for  implementing  a  model.  It  is  the  process  of  conducting 
experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of  the  system  modeled  under 
selected  conditions  or  of  evaluating  various  strategies  for  the  operation  of  the  system  within  the  limits 
imposed  by  developmental  or  operational  criteria.  Simulation  may  include  the  use  of  analog  or  digital 
devices,  laboratory  models,  or  “test  bed”  sites.  Simulations  are  usually  programmed  for  solution  on  a 
computer;  however,  in  the  broadest  sense,  military  exercises,  and  war  games  are  also  simulations. 

SIMULATOR  —  A  generic  term  used  to  describe  equipment  used  to  represent  weapon  systems  in 
development  testing,  operational  testing,  and  training,  e.g.,  a  threat  simulator  has  one  or  more  char¬ 
acteristics  which,  when  detected  by  human  senses  or  man-made  sensors,  provide  the  appearance  of 
an  actual  threat  weapon  system  with  a  prescribed  degree  of  fidelity. 

SPECIFICATION  —  A  document  used  in  development  and  procurement  which  describes  the  techni¬ 
cal  requirements  for  items,  materials,  and  services,  including  the  procedures  by  which  it  will  be 
determined  that  the  requirements  have  been  met.  Specifications  may  be  unique  to  a  specific  pro¬ 
gram  (program-peculiar)  or  they  may  be  common  to  several  applications  (general  in  nature). 

SUBTEST  —  An  element  of  a  test  program.  A  subset  is  a  test  conducted  for  a  specific  purpose  (e.g., 
rain,  dust,  transportability,  missile  firing,  fording). 

SURVIVABILITY  —  Survivability  is  the  capability  of  a  system  and  its  crew  to  avoid  or  withstand  a 
man-made  hostile  environment  without  suffering  an  abortive  impairment  of  its  ability  to  accomplish 
its  designated  mission. 
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SUSCEPTIBILITY  —  The  degree  to  which  a  device,  equipment,  or  weapon  system  is  open  to 
effective  attack  due  to  one  or  more  inherent  weaknesses.  Susceptibility  is  a  function  of  operational 
tactics,  countermeasures,  probability  of  enemy  fielding  a  threat,  etc.  Susceptibility  is  considered  a 
subset  of  survivability. 

SYSTEM  —  1.  The  organization  of  hardware,  software,  material,  facilities,  personnel,  data,  and  ser¬ 
vices  needed  to  perform  a  designated  function  with  specified  results,  such  as  the  gathering  of 
specified  data,  its  processing,  and  delivery  to  users.  2.  A  combination  of  two  or  more  interrelated 
equipment’s  (sets)  arranged  in  a  functional  package  to  perform  an  Operational  function  or  to  satisfy 
a  requirement. 

SYSTEM  ENGINEERING,  DEFENSE  —  A  comprehensive,  iterative  technical  management  pro¬ 
cess  that  includes  translating  operational  requirements  into  configured  systems,  integrating  the  tech¬ 
nical  inputs  of  the  entire  design  team,  managing  interfaces,  characterizing  and  managing  technical 
risk,  transitioning  technology  from  the  technology  base  into  program  specific  efforts,  and  verifying 
that  designs  meet  operational  needs.  It  is  a  life  cycle  activity  that  demands  a  concurrent  approach  to 
both  product  and  process  development. 

SYSTEMS  ENGINEERING  PROCESS  (SEP)  —  A  logical  sequence  of  activities  and  decisions 
transforming  an  operational  need  into  a  description  of  system  performance  parameters  and  a  pre¬ 
ferred  system  configuration. 

SYSTEM  SPECIFICATION  —  States  all  necessary  functional  requirements  of  a  system  in  terms  of 
technical  performance  and  mission  requirements,  including  test  provisions  to  assure  that  all  require¬ 
ments  are  achieved.  Essential  physical  constraints  are  included.  System  specifications  state  the  tech¬ 
nical  and  mission  requirements  of  the  system  as  an  entity. 

SYSTEM  THREAT  ASSESSMENT  (STA)  —  Describes  the  threat  to  be  countered  and  the  pro¬ 
jected  threat  environment.  The  threat  information  must  be  validated  by  the  Defense  Intelligence 
Agency  (DIA)  programs  reviewed  by  the  Defense  Acquisition  Board  (DAB). 

TECHNICAL  EVALUATION  (TECHEVAL)  —  The  study,  investigations,  or  Test  and  Evaluation 
(T&E)  by  a  developing  agency  to  determine  the  technical  suitability  of  materiel,  equipment,  or  a 
system,  for  use  in  the  military  Services.  (See  Development  Test  and  Evaluation  (DT&E),  Navy.) 

TECHNICAL  FEASIBILITY  TEST  —  A  technical  feasibility  test  is  a  developmental  test  typically 
conducted  during  concept  refinement  and  technology  development  to  provide  data  to  assist  in 
determining  safety  and  health  hazards,  and  establishing  system  performance  specifications  and  feasibility. 

TECHNICAL  PERFORMANCE  MEASUREMENT  (TPM)  —  Describes  all  the  activities  un¬ 
dertaken  by  the  government  to  obtain  design  status  beyond  that  treating  schedule  and  cost.  A  TPM 
manager  is  defined  as  the  product  design  assessment  which  estimates,  through  tests  the  values  of 
essential  performance  parameters  of  the  current  design  of  Work  Breakdown  Structure  (WBS)  prod¬ 
uct  elements.  It  forecasts  the  values  to  be  achieved  through  the  planned  technical  program  effort, 
measures  differences  between  achieved  values  and  those  allocated  to  the  product  element  by  the 
system  engineering  process,  and  determines  the  impact  of  these  differences  on  system  effectiveness. 
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TECHNICAL  TESTER  —  The  command  or  agency  that  plans,  conducts  and  reports  the  results  of 
Army  technical  testing.  Associated  contractors  may  perform  development  testing  on  behalf  of  the 
command  or  agency. 

TEST  —  Any  program  or  procedure  which  is  designed  to  obtain,  verify,  or  provide  data  for  the 
evaluation  of  Research  and  Development  (R&D)  (other  than  laboratory  experiments);  progress  in 
accomplishing  development  objectives;  or  performance  and  operational  capability  of  systems,  sub¬ 
systems,  components,  and  equipment  items. 

TEST  AND  EVALUATION  (T&E)  —  Process  by  which  a  system  or  components  provide  informa¬ 
tion  regarding  risk  and  risk  mitigation  and  empirical  data  to  validate  models  and  simulations.  T&E 
permit,  as  assessment  of  the  attainment  of  technical  performance,  specifications  and  system  matu¬ 
rity  to  determine  whether  systems  are  operationally  effective,  suitable  and  survivable  for  intended 
use.  There  are  two  general  types  of  T&E-Developmental  (DT&E)  and  Operational  (OT&E).  (See 
Operational  Test  and  Evaluation  (OT&E),  Initial  Operational  Test  and  Evaluation  (IOT&E),  and 
Developmental  Test  and  Evaluation  (DT&E).) 

TEST  AND  EVALUATION  MASTER  PLAN  (TEMP)  —  Documents  the  overall  structure  and 
objectives  of  the  Test  and  Evaluation  (T&E)  program.  It  provides  a  framework  within  which  to 
generate  detailed  T&E  plans  and  it  documents  schedule  and  resource  implications  associated  with 
the  T&E  program.  The  TEMP  identifies  the  necessary  Developmental  Test  and  Evaluation  (DT&E), 
Operational  Test  and  Evaluation  (OT&E)  and  Live  Fire  Test  and  Evaluation  (LFT&E)  activities.  It 
relates  program  schedule,  test  management  strategy  and  structure,  and  required  resources  to:  Criti¬ 
cal  Operational  Issues  (COIs);  critical  technical  parameters;  objectives  and  thresholds  documented 
in  the  Operational  Requirements  Document  (ORD);  evaluation  criteria;  and  (5)  milestone  decision 
points.  For  multi-Service  or  joint  programs,  a  single  integrated  TEMP  is  required.  Component- 
unique  content  requirements,  particularly  evaluation  criteria  associated  with  COIs,  can  be  addressed 
in  a  component-prepared  annex  to  the  basic  TEMP.  (See  Capstone  TEMP).) 

TEST  BED  —  A  system  representation  consisting  of  actual  hardware  and/or  software  and  computer 
models  or  prototype  hardware  and/or  software 

TEST  CRITERIA  —  Standards  by  which  test  results  and  outcome  are  judged. 

TEST  DESIGN  PLAN  —  A  statement  of  the  conditions  under  which  the  test  is  to  be  conducted,  the 
data  required  from  the  test  and  the  data  handling  required  to  relate  the  data  results  to  the  test 
conditions. 

TEST  INSTRUMENTATION  —  Test  instrumentation  is  scientific,  Automated  Data  Processing  Equip¬ 
ment  (ADPE),  or  technical  equipment  used  to  measure,  sense,  record,  transmit,  process  or  display 
data  during  tests,  evaluations  or  examination  of  materiel,  training  concepts  or  tactical  doctrine. 
Audio-visual  is  included  as  instrumentation  when  used  to  support  Army  testing. 

TEST  RESOURCES  — A  collective  term  that  encompasses  all  elements  necessary  to  plan,  conduct 
and  collect/analyze  data  from  a  test  event  or  program.  Elements  include  test  funding  and  support 
manpower  (including  Temporary  Duty  (TDY)  costs),  test  assets  (or  units  under  test),  test  asset 
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support  equipment,  technical  data,  simulation  models,  test-beds,  threat  simulators,  surrogates  and 
replicas,  special  instrumentation  peculiar  to  a  given  test  asset  or  test  event,  targets,  tracking  and  data 
acquisition,  instrumentation,  equipment  for  data  reduction,  communications,  meteorology,  utilities, 
photography,  calibration,  security,  recovery,  maintenance  and  repair,  frequency  management  and 
control,  and  base/facility  support  services. 

THREAT  — The  sum  of  the  potential  strengths,  capabilities,  and  strategic  objectives  of  any  adversary 
that  can  limit  or  negate  U.S.  mission  accomplishment  or  reduce  force,  system,  or  equipment 
effectiveness. 

THRESHOLDS  —  The  minimum  acceptable  value  which,  in  the  user’s  judgment,  is  necessary  to 
satisfy  the  need.  If  threshold  values  are  not  achieved,  program  performance  is  seriously  degraded, 
the  program  may  be  too  costly,  or  the  program  may  no  longer  be  viable. 

TRANSPORTABILITY  —  The  capability  of  materiel  to  be  moved  by  towing,  self-propulsion,  or 
carrier  through  any  means,  such  as  railways,  highways,  waterways,  pipelines,  oceans,  and  airways. 
(Full  consideration  of  available  and  projected  transportation  assets,  mobility  plans  and  schedules, 
and  the  impact  of  system  equipment  and  support  items  on  the  strategic  mobility  of  operating  military 
forces  is  required  to  achieve  this  capability.) 

UNKNOWN-UNKNOWNS  (UNK(s))  —  Future  situation  impossible  to  plan,  predict,  or  even  know 
what  to  look  for. 

USER  —  An  operational  command  or  agency  that  receives  or  will  receive  benefit  from  the  acquired 
system.  Combatant  Commanders  (COCOMs)  and  the  Services  are  the  users.  There  may  be  more 
than  one  user  for  a  system.  The  Services  are  seen  as  users  for  systems  required  to  organize,  equip, 
and  train  forces  for  the  COCOMs  of  the  unified  command. 

USER  FRIENDLY  —  Primarily  a  term  used  in  Automated  Data  Processing  (ADP),  it  connotes  a 
machine  (hardware)  or  program  (software)  that  are  compatible  with  a  person’s  ability  to  operate 
them  successfully  and  easily. 

USER  REPRESENTATIVES  —  A  command  or  agency  that  has  been  formally  designated  by  proper 
authority  to  represent  single  or  multiple  users  in  the  requirements  and  acquisition  process.  The 
Services  and  the  Service  components  of  the  Combatant  Commanders  (COCOMs)  are  normally  the 
user  representative.  There  should  be  only  one  user  representative  for  a  system. 

VALIDATION  —  1.  The  process  by  which  the  contractor  (or  as  otherwise  directed  by  the  DoD 
component  procuring  activity)  tests  a  publication/Technical  Manual  (TM)  for  technical  accuracy 
and  adequacy.  2.  The  procedure  of  comparing  input  and  output  against  an  edited  file  and  evaluat¬ 
ing  the  result  of  the  comparison  by  means  of  a  decision  table  established  as  a  standard. 

VARIANCE  (Statistical)  —  A  measure  of  the  degree  of  spread  among  a  set  of  values;  a  measure  of 
the  tendency  of  individual  values  to  vary  from  the  mean  value.  It  is  computed  by  subtracting  the 
mean  value  from  each  value,  squaring  each  of  these  differences,  summing  these  results,  and  divid¬ 
ing  this  sum  by  the  number  of  values  in  order  to  obtain  the  arithmetic  mean  of  these  squares. 
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VULNERABILITY  —  The  characteristics  of  a  system  that  cause  it  to  suffer  a  definite  degradation 
(loss  or  reduction  of  capability  to  perform  the  designated  mission)  as  a  result  of  having  been  subjected 
to  a  certain  (defined)  level  of  effects  in  an  unnatural  (man-made)  hostile  environment.  Vulnerability 
is  considered  a  subset  of  survivability. 

WORK  BREAKDOWN  STRUCTURE  (WBS)  —  An  organized  method  to  break  down  a  project 
into  logical  subdivisions  or  subprojects  at  lower  and  lower  levels  of  details.  It  is  very  useful  in 
organizing  a  project. 

WORKING-LEVEL  INTEGRATED  PRODUCT  TEAM  (WIPT)  —  Team  of  representatives  from 
all  appropriate  functional  disciplines  working  together  to  build  successful  and  balanced  programs, 
identify  and  resolve  issues,  and  make  sound  and  timely  decisions.  WIPTs  may  include  members 
from  both  government  and  industry,  including  program  contractors  and  sub-contractors.  A  commit¬ 
tee,  which  includes  non-government  representatives,  to  provide  an  industry  view,  would  be  an 
advisory  committee  covered  by  Federal  Advisory  Committee  Act  (FACAj  and  must  follow  the 
procedures  of  that  Act. 
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APPENDIX  C 
TEST  RELATED  DATA 
ITEM  DESCRIPTIONS 


extracted  from  DoD  5010.12-L, 
Acquisition  Management  System  and 
Data  Requirement  Control  List  (AMSDL) 


Acceptance  Test  Plan 

DI-QCIC-80154,  80553 

Airborne  Sound  Measurements  Test  Report 

DI-HFAC-80272 

Airframe  Rigidity  Test  Report 

DI-T-30734 

Ammunition  Test  Expenditure  Report 

DI-MISC-80060 

Armor  Material  Test  Reports 

DI-MISC-80073 

Ballistic  Acceptance  Test  Report 

DI-MISC- 80246 

C.P.  Propeller  Test  Agenda 

UDI-T-23737 

Coordinated  Test  Plan 

DI-MGMT-80937 

Corrosion  Testing  Reports 

DI-MFFP-80108 

Damage  Tolerance  Test  Results  Reports 

DI-T-30725 

Demonstration  Test 

Plan 

DI-QCIC- 80775 

Report 

DI-QCIC-80774 

Directed  Energy  Survivability  Test  Plan 

DI-R-1786 

Durability  Test  Results  Report 

DI-T-30726 

Electromagnetic  Compatibility  Test  Plan 

DI-T-3704B 

Electromagnetic  Interference  Test 

Plan 

DI-EMCS-80201 

Report 

DI-EMCS-80200 

Electrostatic  Discharge  Sensitivity  Test  Report 

DI-RELI-80670 

Emission  Control  (EMCON)  Test  Report 

DI-R-2059 

Endurance  Test  (EMCS)  Failure  Reports 

DI-ATTS-80366 

Engineer  Design  Test  Plan 

DI-MGMT- 8068 8 

Environmental  Design  Test  Plan 

DI-ENVR-80861 

Environmental  Test  Report 

DI-ENVR- 80863 

Equipment  Test  Plan  (Nonsystem) 

DI-T-3709A 

Factory  Test 

Plan 

DI-QCIC-80153 

EMCS  Plan 

DI-ATTS-80360 

EMCS  Procedures 

DI-ATTS-80361 

EMCS  Reports 

DI-ATTS-80362 

First  Article  Qualification  Test  Plan 

DI-T-5315A 

Flight  Flutter  Test  Report 

DI-T-30733 

Flutter  Model  Test  Report 

DI-T-30732 

Hardware  Diagnostic  Test  System  Development  Plan 

DI-ATTS-80005 

High-Impact  Shock  Test  Procedures 

DI-ENVR-80709 

Hull  Test  Results  (Boats)  Report 

UDI-T-23718 
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Human  Engineering  Test 
Plan 
Report 

Inspection  and  Test  Plan 
Installation  Test 
Plan 

Procedures 

Report 

Integrated  Circuit  Test  Documentation 
Maintainability/Testability  Demonstration  Test 
Plan 
Report 

Maintenance  Training  Equipment  Test  Outlines 
Master  Test  Plan/Program  Test  Plan 
NBC  Contamination  Survivability  Test  Plan 
Nuclear  Survivability  Test 
Plan 
Report 
Packaging  Test 
Plan 
Report 

Part,  Component,  or  Subsystem  Test  Plan(s) 

Parts  (Non-standard)  Test  Data  Report 
Parts  Qualification  Test  Plan 
Performance  Oriented  Packaging  Test  Report 
Production  Test 
Plan 
Report 

Quality  Conformance  Test  Procedures 
Radar  Spectrum  Management  (RSM)  Test  Plan 
Randomizer  Test  Report 
Reliability  Test 
Plan 

Procedures 

Reports 

Research  and  Development  Test  and  Acceptance  Plan 
Rough  Handling  Test  Report 
Ship  Acceptance  Test  (SAT) 

Schedule 

Report 

Shipboard  Industrial  Test  Procedures 
Shock  Test 

Extension  Request 
Report 

Software  General  Unit  Test  Plan 
Software  Test 
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DI-HFAC-80743 

DI-HFAC-80744 

DI-QCIC-81110 

DI-QCIC-80155 
DI-QCIC-805 1 1 
DI-QCIC-80140,  80512 
DI-ATTS-80888 

DI-MNTY-8083 1 

DI-MNTY-80832 

DI-H-6129A 

DI-T-30714 

DI-R-1779 

DI-NUOR-80928 

DI-NUOR-80929 

DI-PACK-80456 

DI-PACK-80457 

DI-MISC-80759 

DI-MISC-81058 

DI-T-5477A 

DI-PACK-81059 

DI-MNT  Y-  80173 

DI-NDTI-80492 

DI-RELI-80322 

DI-MISC-81113 

DI-NDTI-80884 

DI-RELI-80250 
DI-RELI-8025 1 
DI-RELI-80252 
DI-T-30744 
DI-T-5144C 

DI-T-23959B 

DI-T-23190A 

DI-QCIC-80206 

DI-ENVR-80706 

DI-ENVR-80708 

DI-MCCR-80307 


Description 

DI-MCCR-80015A 

Plan 

DI-MCCR- 800 1 4 A 

Procedures 

DI-MCCR-80310 

Report 

DI-MCCR- 800 1 7 A,  80311 

Software  System 

Devel  Test  and  Eval  Plan 

DI-MCCR- 80309 

Integration  and  Test  Plan 

DI-MCCR- 80308 

Sound  Test  Failure  Notif  and  Recomm 

DI-HFAC-80271 

Special  Test  Equipment  Plan 

DI-T-30702 

Spectrum  Signature  Test  Plan 

DI-R-2068 

Static  Test 

Plan 

DI-T-21463A 

Reports 

DI-T-21464A 

Structureborne  Vibration  Accel  Measurement  Test 

DI-HFAC-80274 

Superimposed  Load  Test  Report 

DI-T-5463A 

Tempest  Test 

Request 

DI-EMCS-80218 

Plan 

DI-T-1912A 

Test  Change  Proposal 

DI-T-26391B 

Test  Elements  List 

DI-QCIC-80204 

Test  Facility  Requirements  Document  (TFRD) 

DI-FACR-80810 

Test  Package 

DI-ILSS-81085 

Test 

Plan 

DI-NDTI- 80566 

Plans/Procedures 

DI-NDTI-80808 

Procedure 

DI-NDTI-80603 

Procedures 

UDI-T-23732B 

Test  Plan  Documentation  for  AIS 

DI-IPSC-80697 

Test  Program 

Documentation  (TPD) 

DI-ATTS-80284 

Integration  Logbook 

DI-ATTS-80281 

TPS  and  OTPS  Acceptance  Test 

Procedures  (ATPS) 

DI-ATTS-80282A 

Report  (ATR) 

DI-ATTS-80283A 

Test  Reports 

DI-NDTI-80809A, 

DI-MISC-80653 

Test  Requirements  Document 

DI-T-2181, 

DI-ATTS-80002,  80041 

Test  Scheduling  Report 

DI-MISC-80761 

Testability 

Program  Plan 

DI-T-7198 

Analysis  Report 

DI-T-7199 

Trainer  Test  Procedures  and  Results  Report 

DI-T-25594C 

Vibration  and  Noise  Test  Reports 

DI-T-30735 
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Vibration  Testing 
Extension 
Report 

Welding  Procedure  Qualification  Test  Report 


UDI-T-23752 

UDI-T-23762 

DI-MISC-80876 
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APPENDIX  D 
BIBLIOGRAPHY 

DEPARTMENT  OF  DEFENSE  DIRECTIVES  (DoDD):  Many  of  the  references  used  in  the  first 
edition  were  cancelled  and  their  intent  incorporated  in  the  5000  Series  documents  issued  23  October 
2000  and  implementing  Service  documents. 

1 .  DoDD  3200. 1 1 ,  Major  Range  and  Test  Facility  Base  (MRTFB). 

2.  DoD  3200. 1 1  -D,  Major  Range  and  Test  Facility  Base  Summary  of  Capabilities. 

3.  (Cancelled)  DoDD  4245.6,  Defense  Production  Management. 

4.  (Cancelled)  DoDD  4245.7,  Transition  from  Development  to  Production. 

5.  DoDD  5000.1,  The  Defense  Acquisition  System. 

6.  (Cancelled)  DoDD  5000  .3,  Test  and  Evaluation. 

7.  (Cancelled)  DoDD  5000.37,  Acquisition  and  Distribution  of  Commercial  Products 

(ADCOP). 

8.  (Cancelled)  DoDD  5000.38,  Production  Readiness  Reviews. 

9.  (Cancelled)  DoDD  5000.39,  Acquisition  and  Management  of  Integrated  Logistic  Support 

for  Systems  and  Equipment. 

10.  DoDD  5010.8,  Value  Engineering  Program. 

1 1 .  (Cancelled)  DoDD  5 105.3 1 ,  Defense  Nuclear  Agency. 

12.  DoDD  5134.1,  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 

( USD(AT&L )). 

1 3 .  DoDD  5 1 4 1 .2,  Director  of  Operational  Test  and  Evaluation  ( DOT &E ). 

14.  DoDD  5160.5,  Responsibilities  for  Research,  Development,  and  Acquisition  of  Chemical 

Weapons  and  Chemical  and  Biological  Defense. 

DEPARTMENT  OF  DEFENSE  INSTRUCTIONS 

15.  (Cancelled)  DoDI  4245.4,  Acquisition  of  Nuclear-Survivable  Systems. 
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16.  DoDI  5000.2,  Operation  of  the  Defense  Acquisition  System. 

17.  DoDI  5030.55,  Joint  AEC-DoD  Nuclear  Weapons  Development  Procedures. 

18.  (Cancelled)  DoDI  7000.10,  Contractor  Cost  Performance,  Funds  Status,  and  Cost/ 

Schedule  Reports. 

DEPARTMENT  OF  DEFENSE  GUIDES/MANUALS 

19.  DoD  4245.7-M,  Transition  from  Development  to  Production. 

20.  A.  (Cancelled)  DoD  5000.3-M-l,  Test  and  Evaluation  Master  Plan  (TEMP)  Guidelines. 

20.  B.  DoD  5000.3-M-2,  Foreign  Weapons  Evaluation  Program  and  NATO  Comparative  Test 

Program  Procedures  Manual. 

21.  (Cancelled)  DoD  5000.3-M-3,  Software  Test  and  Evaluation  Manual. 

DEPARTMENT  OF  DEFENSE  STANDARDS 

22.  DoD-STD-2167,  Defense  System  Software  Development. 

23.  DoD-STD-2168,  Defense  System  Software  Quality  Program. 

MILITARY  STANDARDS 

24.  MIL-STD-480,  Configuration  Control  -  Engineering  Changes,  Deviations,  and  Waivers. 

25.  MIL-STD-483  (USAF),  Configuration  Management  Practices  for  Systems,  Equipment, 

Munitions  and  Computer  Software. 

26.  MIL-STD-490,  Specification  Practices. 

27.  MIL-STD-499A  (USAF),  Engineering  Management  Practices. 

28.  MIL-STD- 1388-1  A,  Logistic  Support  Analysis. 

29.  MIL-STD- 1 5 12A,  Technical  Reviews  and  Audits  for  Systems,  Equipment,  and  Computer 

Programs. 

30.  MIL-STD- 1 528,  Manufacturing  Master  Plan. 
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MILITARY  HANDBOOKS  ““ 

3 1 .  Joint  Test  Director’s  Handbook  (Draft). 


OTHER  MILITARY  DOCUMENTS 

32.  Test  and  Evaluation  in  the  United  States  Department  of  Defense,  Office  of  the  Under 

Secretary  of  Defense  for  Research  and  Engineering  (Director,  Defense  Test  and 
Evaluation),  August  1983. 

33.  Statement  by  Mr.  James  F.  O’Bryon,  Assistant  Deputy  Under  Secretary  of  Defense  (Live 

Fire  Test)  before  the  Acquisition  Policy  Panel  of  the  House  Armed  Services  Committee, 
10  September  1987. 

34.  Memorandum  of  Agreement  of  Multi-Service  OT&E,  with  changes  (August  2003). 

35.  Joint  Test  and  Evaluation  Procedures  Manual,  (Draft),  Office  of  the  Deputy  Under 

Secretary  for  Research  and  Engineering  (Test  and  Evaluation),  October  1986. 

36.  Live  Fire  Test  and  Evaluation  Planning  Guide,  Office  of  the  Deputy  Director,  Test  and 

Evaluation  (Live  Fire  Test),  June  1989. 

37.  Joint  Logistics  Commanders  Guidance  for  the  Use  of  an  Evolutionary  Acquisition  (EA) 

Strategy  in  Acquiring  Command  and  Control  (C2)  Systems,  Defense  Systems 
Management  College,  March  1987. 

38.  Concept  and  Approach  for  a  Joint  Test  and  Evaluation  of  U.S.  Chemical  Warfare 

Retaliatory  Capabilities,  JCHEM  Joint  Test  Force,  AD  #B088340,  February  1984. 

39.  FY88  Report  of  the  Secretary  of  Defense  to  the  Congress. 

40.  Report  of  Task  Force  on  Test  and  Evaluation,  Defense  Science  Board,  1  April  1974. 

41 .  Solving  the  Risk  Equation  in  Transitioning  from  Development  to  Production,  Defense 

Science  Board  Task  Force  Report,  25  May  1983  (later  published  as  DoD  Manual 
4245.7). 

42.  Risk  Management  as  a  Means  of  Direction  and  Control,  Program  Manager’s  Notebook, 

Fact  Sheet  Number  4.5,  Defense  Systems  Management  College,  June  1992. 

43.  Joint  Logistics  Commanders  Guide  for  the  Management  of  Joint  Service  Programs, 

Defense  Systems  Management  College,  1987. 

44.  Systems  Engineering  Management  Guide,  Second  Edition,  Defense  Systems  Management 

College,  1990. 
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45.  Integrated  Logistics  Support  Guide,  First  Edition,  Defense  Systems  Management  College, 

May  1986. 

46.  Joint  Logistics  Commanders  Guide  for  the  Management  of  Multinational  Programs, 

Defense  Systems  Management  College,  1987. 

47.  Defense  Manufacturing  Management  Course,  Defense  Systems  Management  College. 

48.  DVAL  Methodology  —  Methodology  Overview/Executive  Summary,  Data  Link 

Vulnerability  Analysis  Joint  Test  Force,  November  1984. 

ARMY  DOCUMENTS 

49.  (Cancelled)  AR  10-4,  U.S.  Army  Operational  Test  and  Evaluation  Agency. 

50.  AR  10-11,  U.S.  Army  Materiel  Development  and  Readiness  Command. 

51.  AR  10-41,  U.S.  Army  Training  and  Doctrine  Command. 

52.  AR  10-42,  U.S.  Armed  Forces  Command. 

53.  AR  70-24,  Special  Procedures  Pertaining  to  Nuclear  Weapons  System  Development. 

54.  AR  70-60,  Nuclear  Survivability  of  Army  Materiel. 

55.  AR  70-71,  Nuclear,  Biological,  and  Chemical  Contamination  Survivability  of  Army 

Materiel. 

56.  AR  71-1,  System  Acquisition  Policy  and  Procedures  (replaced  by  AR  70-1). 

57.  AR  71-3,  Force  Development  User  Testing. 

58.  AR  73-1,  Test  and  Evaluation  Policy. 

59.  AR  700-127,  Integrated  Logistic  Support. 

60.  DA  PAM  73-1,  Test  and  Evaluation  in  Support  of  System  Acquisition. 

61.  (Cancelled)  DA  PAM  700-50,  Integrated  Logistic  Support:  Developmental  Supportability 

Test  and  Evaluation  Guide. 

62.  AMC  Plan  70-2,  AMC-TRADOC  Materiel  Acquisition  Handbook. 

63.  TECOM  PAM  310-4,  Index  of  Test  Operations  Procedures  and  International  Test 

Operations  Procedures. 
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64.  (Cancelled)  USAOTEA  Operational  Test  and  Evaluation  Guide,  Special  Supplement, 

Continuous  Comprehensive  Evaluation  Planning  for  Nuclear  Hardness  and  Survivability 
Test  and  Evaluation  (Draft). 

NAVY  DOCUMENTS 

65 .  SECNAVINST  5000. 1 B ,  System  Acquisition. 

66.  SECNAVINST  5000.39,  USN  Acquisition  and  Management  of  ILS  Systems  and 

Equipment. 

67.  SECNAVINST  5000.42C,  RDT&E  Acquisition  Procedures. 

68.  SECNAVINST  5430.67 A,  Assignment  of  Responsibilities  for  Research,  Development,  Test 

and  Evaluation. 

69.  OPNAVINST  3070.1 A,  Operations  Security. 

70.  (Cancelled)  OPNAVINST  3960. 10C,  Test  and  Evaluation  (Draft). 

7 1 .  OPNAVINST  5000.49,  ILS  in  the  Acquisition  Process. 

12.  OPNAVINST  5440.47F,  Mission  and  Functions  of  Operational  Test  and  Evaluation  Force 
(OPTEVFOR). 

73.  NAVMATINST  3960,  Test  and  Evaluation. 

14.  COMOPTEVFORINST  3960.  ID,  Operational  Test  Director’s  Guide. 

15.  NAVSO  P-2457,  RDT&E/ Acquisition  Management  Guide,  10th  Edition. 

76.  COMOPTEVFOR,  Project  Analysis  Guide. 

11.  COMOPTEVFOR,  AD  Number  AD124168,  Operational  Test  Director  Guide. 

AIR  FORCE  DOCUMENTS  "  — — 

78.  AFR  23-36,  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC). 

19.  AFR  55-30,  Operations  Security. 

80.  AFM  55-43,  Management  of  Operational  Test  and  Evaluation. 

81.  (Cancelled)  AFR  80-14,  Test  and  Evaluation. 
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82.  AFR  80-38,  Management  of  the  Air  Force  Systems  Survivability  Program. 

83.  AFR  800-2,  Acquisition  Program  Management. 

84.  AFR  800-8,  Integrated  Logistic  Support  (ILS)  Program. 

85.  CAFSCR  550-15,  Statistical  Process  Control. 

86.  AFSCR  550-13,  Producibility. 

87.  AFOTEC  Regulation  23-1,  Organization  and  Functions  Air  Force  Operational  Test  and 

Evaluation  Center  (AFOTEC). 

88.  AFOTEC,  Nuclear  Survivability  Handbook  for  Air  Force  OT&E. 

89.  AFOTEC,  Electronic  Warfare  Operational  Test  and  Evaluation. 

90.  AFLC  Pamphlet  800-34,  ACQ  Logistic  Management. 

9 1 .  Rome  Air  Development  Center,  Standard  Procedures  for  Air  Force  Operational  Test  and 

Evaluation. 

MARINE  CORPS  DOCUMENTS 

92.  MCO  3960.2,  Marine  Corps  Operational  Test  and  Evaluation  Activity  (MCOTEAf 

establishment  of,  with  changes. 

93.  MCO  5000.1  IB,  Testing  and  Evaluation  of  Systems  and  Equipment  for  the  Marine  Corps. 

OTHER  GOVERNMENT  DOCUMENTS 

94.  Public  Law  99-661,  National  Defense  Authorization  Act  for  Fiscal  Year  1987. 

95.  Public  Law  92-156. 

96.  Public  Law  98-94. 

97.  GAO/PEMD-87-17,  Live  Fire  Testing:  Evaluating  DoD’s  Programs. 

98.  GAO/NSIAD-87-57,  Operational  Test  and  Evaluation  Can  Contribute  More  to  Decision- 

Making. 

99.  GAO/PEMD-86-12BR,  Bigeye  Bomb:  An  Evaluation  of  DoD’s  Chemical  and 

Developmental  Tests. 
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100.  GAO/PS  AD-79-86,  Review  of  Military  Services  Development  Test  and  Evaluation  of  Six 

Weapon  Systems  Totaling  an  Estimated  $12  Billion  in  Development  and  Procurement 
Costs. 

101.  GAO/PS  AD-78- 102,  Operational  Testing  of  Air  Force  Systems  Requires  Several 

Improvements. 

102.  Memorandum  of  Understanding  Among  the  Governments  of  the  French  Republic,  the 

Federal  Republic  of  Germany,  the  United  Kingdom  of  Great  Britain  and  Northern 
Ireland,  and  the  United  States  of  America  Relating  to  the  Mutual  Acceptance  of  Test  and 

Evaluation  for  the  Reciprocal  Procurement  of  Defense  Equipment. 

103.  Director,  Defense  Test  and  Evaluation  Memorandum  of  August  2,  1983;  Response  of 

Admiral  Isham  Linder,  Director,  Test  and  Evaluation  to  Supplemental  Questions  posed  by 
the  Senate  Governmental  Affairs  Committee. 

104.  Hearing  before  the  Senate  Committee  on  Governmental  Affairs,  June  23, 1983. 


OTHER  REFERENCES 

105.  Anderson,  J.E.,  “Towards  the  Goal  of  Improving  MIL-STD  Testing,”  Defense  Systems 

Management  Journal,  Vol.  12,  Number  2,  pages  30-34,  April  1976. 

106.  The  BDM  Corporation,  Application  of  Simulations  to  the  OT&E  Process,  May  1974. 

107.  Bolino,  John  V.,  Software  T&E  in  the  Department  of  Defense. 

108.  The  BDM  Corporation,  Functional  Description  of  the  Acquisition  Test  and  Evaluation 

Process,  July  8,  1983. 

109.  Hoivik,  Thomas  H.,  The  Navy  Test  and  Evaluation  Process  in  Major  Systems  Acquisition, 
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110.  Johnson,  Larry  H.,  Contractor  Testing  and  the  Army  Test  and  Evaluation  Master  Plan  for 

Full  Engineering  Development,  Defense  Systems  Management  College,  Fort  Belvoir, 
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111.  Rhoads,  D.I.,  Systems  Engineering  Guide  for  Management  of  Software  Acquisition,  pages 
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1 12.  The  MITRE  Corporation,  Software  Reporting  Metrics,  November  1985. 

1 13.  Stevens,  Roger  T.,  John  Wiley  &  Sons,  New  York,  Operational  Test  and  Evaluation 
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